Forecasting can be useful, but only if done well

An article in The Economist is partially based on a paper in the Proceedings of the National Academy of Sciences by David Mandel and Alan Barnes. Those researchers

analysed more than 1,500 intelligence forecasts produced by a nameless (but presumably Canadian) agency, covering the period from March 2005 to December 2011. 

Their results suggest that the old joke about “military intelligence” being an oxymoron is unfair. When they compared what the analysts had said with what actually came to pass, they found that the predictions were right about three-quarters of the time. Cynics might wonder if the analysts merely restricted themselves to easy cases, but Dr Mandel and Dr Barnes also found they were good at calibrating their judgments. Events they deemed unlikely did not happen often, whereas those they thought likely occurred frequently. Indeed, if anything they were underselling themselves, tending to err more than necessary on the side of uncertainty. And there was evidence that their skills could be learnt—for more-experienced analysts tended to do better than their junior counterparts........

Unlike pundits, who can pontificate from the safety of their armchairs, analysts know that their advice is likely to have consequences in the real world. Drs Mandel and Barnes found that analysts’ inherent underconfidence became even more pronounced when confronting particularly important or difficult questions.

Analysts must also defend their claims to managers, who are trained to be sceptical, and to their political masters. Other studies have shown such accountability encourages careful thinking and reduces self-serving cognitive biases.

The Economist article contrasts this research with

a famous earlier finding. In 2005 Philip Tetlock, a management theorist at the University of Pennsylvania, announced the results of a 20-year study in which 284 experts—professors, journalists, civil servants and so forth—were invited to make more than 28,000 predictions. Their performance was abysmal: barely better than chance, and inferior even to simple computer algorithms......... Dr Tetlock found that the more famous his pundits were, the worse they did.

These data are worth keeping in mind. The talking heads one sees on television and the prognostications one reads in the media (or worse in blogs) are often wrong. Intelligence agencies often do a lot better! President Obama is probably working from much better forecasts than those that might be available to you.