When experts argue

In 1984, a young psychologist called Philip Tetlock began an extraordinary two-decade investigation into the limits of expertise. As the most junior member of a committee of the National Academy of Sciences, Tetlock was charged with figuring out how social scientists might help interpret the implications of Ronald Reagan’s hawkish stance in the cold war. He canvassed every expert he could find, and was struck by the fact that the most influential thinkers on the cold war flatly contradicted each other.

Tetlock’s resulting book, Expert Political Judgment, is excellent but technical. It has become influential: you can read accounts of it in Nassim Nicholas Taleb’s The Black Swan, Dan Gardner’s Future Babble and Kathryn Schulz’s Being Wrong. Dan Gardner’s book asks the logical next question: if, as Tetlock shows, expert forecasts are such a disappointment, why do we seem to be so addicted to them? It is a puzzle. Although we know forecasters are often humbled, it doesn’t stop our strange fascination with forecasting: we love to be told, confidently, what will happen next.
Disagreement between talking heads seems to be the natural order of the universe, but Tetlock wasn’t content to leave it at that: after all, when the leading experts simply disagree about the key problem of the age, what should that tell us about the limits of expertise? So he rounded up nearly 300 experts, from a variety of professions and academic disciplines, asked them to make specific, quantifiable forecasts – they answered 27,450 questions between them – and then waited to see whether the forecasts came true. They rarely did. [Apologies - this paragraph was missing. T.H.]

I am often asked to make forecasts myself, to my bafflement. The honest answer to “will the cuts cause a double-dip recession?” or “will the eurozone survive?” is “I don’t know”. Perhaps I should just offer a prophecy instead – after all, says Gardner, nobody is likely to catch me out. We don’t follow up on forecasts. We assume correct forecasts are the result of brilliance rather than luck, and we fail to call people on forecasts-gone-wrong. We are also happy to accept feeble excuses: the forecast almost came true; or the forecast will come true eventually. (One of the secrets of Tetlock’s study was simply that he kept careful records and wouldn’t let his experts wriggle away from failed forecasts.)

No wonder economists make forecasts: we are supplying what the market demands.To the New Yorker’s Louis Menand, “the best lesson of Tetlock’s book may be the one that he seems most reluctant to draw: Think for yourself.” It’s a sound enough principle but there’s a reason why Tetlock himself hesitates to draw that conclusion: his results clearly show that experts outperform non-experts.

To my mind, the problem is not the experts. It’s that the world is simply too complicated for anyone to analyse with much success. One response to that is not to demand forecasts when forecasts are useless. But that approach only goes so far, because most decisions in life involve some element of forecasting. George Osborne’s austerity budgets, Steve Jobs’s “walled garden” strategy for Apple, my decision not to study for a degree in English literature – all of these choices required a view about what the future holds.

That means that our plans will often need to be torn up. This doesn’t sit easily with many voters, or shareholders. Margaret Thatcher famously said “the lady’s not for turning”, while Tony Blair proclaimed that he didn’t have a reverse gear. These would not be attractive attributes in cars, but they seemed to go down well enough with voters, who rewarded the pair with six general election victories between them. Yet if the road ahead is unknowable, the ability to change direction should not be underrated.

Also published at ft.com.