The risky business of forecasting

(First published: 20091016 / Last Updated: 20131116) —

Forecasting the future is a difficult task, as there are many more ways to be wrong than right! In addition, a forecast is not just a statement about what might happen, it is also embedded in a context that may be as relevant as the forecast itself when it comes to assessing its credibility and uptake. The focus is on natural disasters such as earthquakes; it is shown that next to extreme geophysical factors, equally real legal risks may also be lurking  in the background for the forecasters. For a note on the broader issue of forecasting the future of our society, see News from the future: how good are our forecasts?

Mieux vaut prévoir sans certitude que de ne pas prévoir du tout (It is far better to foresee without certainty than not to foresee at all) – Henri Poincaré, La Science et l’hypothèse, IV Partie, La Nature, Chapitre IX, 1902. The full text of La science et l’Hypothèse available here).
Il vaut mieux employer notre esprit à supporter les infortunes qui nous arrivent qu’à prévoir celles qui nous peuvent arriver ( It is a better use for our mind to endure misfortunes that happen to us than to foresee those that may happen) – François de La Rochefoucauld, Maxime 174, Maximes (5 editions from 1665 to 1678). Full text from Project Gutenberg.
With apologies to Tennyson, ’tis better to have explored and lost than never to have explored at all. Scott E. Page and John H. Miller. 2007. Complex Adaptive Systems: An Introduction to Computational Models of Social Life (Princeton Studies in Complexity). Kindle location 585.
Source: http://en.wikipedia.org/wiki/2009_L%27Aquila_earthquake

Source: Wikipedia

We all remember the recent (20090406) earthquake in l’Aquila in central Italy, which was so ably recycled by Silvio Berlusconi for national and international political purposes and image polishing!

According to the Wikipedia article, this earthquake was caused by movement on a NW-SE trending normal fault according to moment tensor solutions. Although Italy lies in a tectonically complex region, the central part of the Appenines has been characterised by extensional tectonics since the Pliocene epoch (i.e. about the last 5 million years), with most of the active faults being normal in type and NW-SE trending. (links removed)

Now this is not the cause, it is the mechanisms with some history. A mechanism, even with some history, is not sufficient an explanation why the quake happened. Listing mechanisms instead of causes is a very common in the geosciences, for instance in meteorology, where we are told that the heavy rain occurred “because high pressure fields moved north or south and a low over central Europe did this or this.” Again, this does not tell why it happened but how it came about.

If we exclude mystical explanations of various kinds (1),  the real causes of the event being forecast should ideally be known. Of course, it is understandable that the final cause is difficult to trace, even with a lot of history; the “real” cause is linked to the event by a chain of mechanisms of the type we are presented with as the “cause”. Nevertheless, a honest attempt should be made to trace back the chain of causalities up to the point where causes still play a part… There is a famous statement by Cicero (see foot note 2) regarding  our ignorance about the causes “which only gods can know”…This is why we humans  use whatever “signs” we have at our disposal. But those signs are no causes, they are indicators. Maybe the “rationale” in the Roman mind is that “everything is connected with everything?”. Or maybe the gods do send us messages , as when the Romans interpreted the flight of birds? I find it a bit intriguing to understand early “warning signals” as “signs”. Scheffer et al. (2009) argue that complex systems may “emit signals” that indicate that a tipping point is about to be reached.  The signals in this case, are unusaul values of variables, or unusual combinations of values.

Let’s come back to the earthquake in l’Aquila: there is an interesting story about a forecast issued in the days and hours before the event  by Giampaolo Giuliani, a research technician (“tecnico di ricerca”)  at the National Physical Laboratory of Gran Sasso. As stated by the Slashdot website, “Giuliani, it turns out, was partially right”. The problem is that a forecast that is only “partially right” is best called “wrong”.  Providing forecasts with lots of estimates about uncertainty of the magnitude and probabilities of occurrence is rarely useful! It also seems that the science behind Giuliani’s forecast was not foolproof, so that, interestingly, we can add it to the many examples of a “correct” forecasts that are the product of chance.  This should not come as a surprise. The discussion on earthquake prediction has been going on for more than 100 years — “both within the scientific community and in its relation with the public — [it] has always been tumultuous.” (Scholtz, 2010)

Sometimes I lie awake at night, and I ask, “Where have I gone wrong?” Then a voice says to me, “This is going to take more than one night.” (Charlie Brown, Charles Schulz’ Peanuts)

Forecasts are difficult because there are many ways in which they can be wrong, but only one way to be correct, i.e. on target regarding it’s occurrence (yes/no), location (coordinates), timing and intensity (all with a range of uncertainty, i.e. the “target” can be large, or small!). Then we have the fact that actual forecasts are a mix of art, science and psychology, plus minor doses of a couple of other ingredients, such as common sense, prudence, guts. For instance, had Giuliani been the director rather than a technician at the National Physical Laboratory of Gran Sasso, his forecasts would certainly have been “less wrong”, even if not necessarily “more correct.” It remains that forecasts are rarely “neutral” when they entail immediate consequences for other people, from slight inconvenience to outright suffering. A well known example is the alarm raised in 1976 by French scientist Claude Allegre, who later became minister, about the imminent eruption of  the Soufriere volcano on the island of Guadeloupe. Haroun Tazieff argued against the evacuation of  75,000 people and  turned out to be right – or luckier – La soufrière controversy illustrates the complex nature of some forecasts: it was not a clash of solid scientic assessments, but a clash of two strong personalities!

Back to l’Aquila: the association of  the victims of the earthquake filed a court case against six members of the Committee for Great Risks and a member of the government who failed to issue a warning (Cartlidge, 2011; Nosengo, 2011) . In fact, the seven defendants are not accused of having issued a wrong forecast. One of them (in the presence of the others, who did not deny the point) stated in a press conference shortly before the events that the risk was actually decreasing because some recent precursor tremors (one of which at the rather high magnitude of 4) had actually released some of the stored energy. This may have encouraged very scared people to stay in their houses instead of taking shelter elsewhere. The specific legal risk incurred by some forecasters is not life threatening but nevertheless very real! The first hearings took place in September 2011 (Panorama and The Economist). The case was   watched closely by many forecasters world wide, as witnessed by the wide international press coverage. At the end of October 2012, six scientists and a former government official were sentenced to six years in prison, after having been found guilty of multiple manslaughter (see BBC, IHT, La Stampa, La Repubblica.) Apart from populism, it’s difficult to understand what is  the deep motivation of the judges. What about the people who did not respect building norms? They get away with it! Let’s wait for the appeal, but be prepared to wait long: for the Vajont disaster of 1963, the final court settlement took place in 1997, 34 years later !

As sometimes happens, the Vatican has a very balanced and measured position, taken from  the address Benedict XVI delivered on 20061106 to the members of the Pontifical Academy of Sciences on the occasion of their plenary assembly held in Rome: scientific predictability also raises the question of the scientist’s ethical responsibilities. His conclusions must be guided by respect for truth and an honest acknowledgment of both the accuracy and the inevitable limitations of the scientific method. Certainly this means avoiding needlessly alarming predictions when these are not supported by sufficient data or exceed science’s actual ability to predict. But it also means avoiding the opposite, namely a silence, born of fear, in the face of genuine problems.

Back to the safer “technical” point of view:  there are many different approaches to forecasting. As I have written elsewhere, forecasting methods can be subdivided into various categories according to the relative share of judgement, statistics, models and data used in the process. But the most common and most obvious method is based on the fact that “things don’t change overnight”, i.e. all systems we deal with have a marked inertia and resist change, so that the safest way of forecasting is to assume that conditions will remain the same, preferably average, and that whatever trends have been underway will continue into the future, at least for “some” time. Easy, but not very exciting or, as the saying goes: no guts, no glory!

The interesting problems arise with drastic changes, fast or violent variations, and surprises, such as extreme events. The word “extreme” itself needs clarifying. It means “rare”, i.e. extreme from a statistical point of view. It does not normally or primarily mean “violent” or “extremely intense”, although the two meanings of “extreme” may overlap. Surprises, by definition, are difficult to forecast because they have a very weak statistical basis, and the usual toolbox of the statistician (levels, trends, seasonal patterns, correlations and autocorrelations) cannot be resorted to.

Nature published an article by a group of scientists who claim even wars can be predicted (See Gilbert, 2009, which is a one-page introduction to Bohorquez et al, 2009). We could probably also quote the recent economic crisis as an example of a major phenomenon that was not forecast with any degree of accuracy. In a recent article in the Italian daily Il Corriere della Sera,  Andrea Ichino, wonders why hurricanes can be forecast, but economic crises cannot.

Wall Street indices predicted nine out of the last five recessions ! (Paul A. Samuelson in Newsweek, Science and Stocks, 19 Sep. 1966. From: Famous forecasting quotes)

He wrongly assumes that one of the reasons is that meteorologists have better data, but he rightly thinks that the second reason is human interference in the form of management and policies. They constitute what Raskin et al. (2002) have categorised as the third type of indeterminacy: volition, next to ignorance and surprise, although I would tend to categorize surprise as a form of ignorance that cannot be modelled with “scientific methods” (Raskin et al., 2002); also look up the internet for “Black Swan Theory” and N.N. Taleb).

Volition also explains why many forecasts that were originally correct become incorrect over time. For instance, if I forecast that 50,000 cows will drown because of floods in the coming weeks, people may take my forecast seriously and move their cattle out of the valley, thereby falsifying the forecast. This is often referred to as “second-order errors” and constitutes the most difficult type of error to deal with in forecasting.

My conclusion is that, in the same way as some animals are more equal than others – and possibly for the same reasons- some forecasts are more reliable than others. The reasons why this is so are complex, and span to whole spectrum from science to  art to politics. In the end, the only way to be reasonably sure of the future is to make it happen!

Notes

(1) According to this AP report (dated 20100419),  A senior Iranian cleric says women who wear revealing clothing and behave promiscuously are to blame for earthquakes. Iran is one of the world’s most earthquake-prone countries, and the cleric’s unusual explanation for why the earth shakes follows a prediction by President Mahmoud Ahmadinejad that a quake is certain to hit Tehran and that many of its 12 million inhabitants should relocate. “Many women who do not dress modestly … lead young men astray, corrupt their chastity and spread adultery in society, which (consequently) increases earthquakes,” Hojatoleslam Kazem Sedighi was quoted as saying by Iranian media.

(2) Marci Tulli Ciceronis de divinatione liber prior, LVI 127:  Qui enim teneat causas rerum futurarum, idem necesse est omnia teneat quae futura sint. Quod cum nemo facere nisi deus possit, relinquendum est homini, ut signis quibusdam consequentia declarantibus futura praesentiat. This translates to Qui tiendrait, en effet, les causes des événements futurs saurait nécessairement quels ils seront. Mais nul autre qu’un dieu n’étant capable de pareille connaissance, il reste que l’homme s’applique à prévoir l’avenir d’après les signes qui le présagent. Source: click here.

References

Bohorquez, J.C, S. Gourley, A.R. Dixon, M. Spagat & N.F. Johnson. 2009. Common ecology quantifies human insurgency. Nature, 462:911-914
Cartlidge, E. 2011. Quake experts to be tried for manslaughter. Science, 332:1135-116
Gilbert, N. 2009. Modellers claim wars are predictable. Nature, 462:836.
Nosengo,N.  2011. Scientists on trial over L’Aquila deaths Seismologists charged for giving apparent reassurances on Italian earthquake. Nature, 404:15.
Raskin, P., T.Banuri,  G.Gallopín, P.Gutman, Al Hammond, R.Kates & R. Swart. 2002. Great transition, the promise and lure of the times ahead. Stockholm Environmental Institute. 99 pp.
Scheffer,M., J.Bascompte, W.A. Brock, V.Brovkin, S.R. Carpenter, V.Dakos, H.Held, E. H.van Nes, M. Rietkerk & G.Sugihara. 2009. Early-warning signals for critical transitions. Nature 461: 53-59.
Scholz,C.H.,  2010. The Prediction Puzzle. Science, 327:1082. A review about “Predicting the unpredictable, the tumultuous science of earthquake prediction”, by Susan Hough.

  5 comments for “The risky business of forecasting

  1. avatar
    Jacques
    20121030 at 18:45

    Un petit commentaire sur les condamnations de l’Aquila en Octobre 2012.

    Il faut accepter à mon sens une certaine culture du risque. Sinon, on arrive à ces absurdes procès -gagnés de plus- que l’on voit aux USA. Mon favori est celui où un couple dîne dans un restaurant, la jeune femme jette son verre à la figure de son compagnon, se lève et glisse dans la flaque qu’elle vient de faire. Le restaurateur fut jugé responsable. Je ne sais pas maintenant, mais de mon temps: années 50-60, les constructions sur la Côte d’Azur n’étaient pas aux normes anti sismiques dans une région qui l’est. Toutes les vieilles maisons ont des croix de fer pour cela. Le calcul est simple pour les entrepreneurs, architectes… il ne se passera rien pendant notre garantie décennale, si oui, on se met en faillite et puis ce sera une catastrophe naturelle et donc nous n’aurons aucune responsabilité… Donc les constructeurs ont intérêt à prendre le risque… et l’acheteur aussi parce que c’est moins cher… je trouve qu’au lycée les exercices de maths et de physique ou d’économie devraient aussi s’attaquer à des exemples très concrets…

  2. avatar
    wergosum
    20091228 at 07:46

    > I really feel stupid because I thought it was important to forecast crop production for my country

    It IS important, even if politicians and the management of your institute do not see the point…yet! They will remember you the next time you have a production crisis!

  3. avatar
    Riad
    20091218 at 21:05

    My personnal experience about crop forecasting :

    15 years ago I was testing what we call crop simulation models, which are pseudo mathematical softwares that reproduce the physiological behaviour of the plants at very small scale. At that time, I was a “young” researcher and all my colleagues were looking at me as a strange agronomist that wants to improve crop production using only his computer. I was fascinated by crop simulation because it was like playing with nature, like god do. I learnt few years after that it was not simple, and I became a crop forecaster. My freind Bernard, who was my thesis supervisor presented me a freind of him who was responsible for the crop forecasting bulletin at EU. I discovered the incredible forecasting machine of the EU. Of course I immediatly understood that EU experience could be reproduced in my country. After a tedious work and some successfull results, last year I produced the first wheat forecasts at national level based on statistical models, beside a crop simulation forecasting done in collaboration with JRC. What was my surprise when I wanted to officially publish the forecasts! All VIPs in my institute were afraid to release them. I understood that I had to publish it on my own risk and that’s what happened. I was lucky as the forecasts were finally correct, but nobody congratulated me and nobody asked me to do it again for the following years! Now, I really feel stupid because I thought it was important to forecast crop production for my country. The moral sense of the story is that we need politicians modelling!

  4. avatar
    20091201 at 18:15

    A statement like “things don’t change overnight” reminds me of the analysis by Schrodinger of the size of atoms.

    Atoms are not small by themselves. They are just small compared to us living objects. The fact might simply be that it is impossible to make a living object out of a few atoms.

    About predictions. It is reasonable to assume that we living things need stability. We would not be able to live if our natural timescale did not match the timescale over which our environment is predictable.

    Rare events might simply be out of our timescale. As such, there is no fundamental reason why they should be predictable at all.

    • avatar
      wergosum
      20091206 at 10:24

      > there is no fundamental reason why they should be predictable at all

      or just the opposite: “things will eventually happen”. It’s just a matter of waiting long enough!

Leave a Reply

Your email address will not be published. Required fields are marked *