Why forecasters failed to predict Trump’s victory
British Remainers watched the US presidential campaign with an uneasy sense that they had seen it all before: brazen lies from a populist movement, experts lining up to attest that all sensible people agree on what should be done … Those of us who saw the EU referendum campaign up close have been well prepared for the possibility of a Trump victory. US Democrats had less of a visceral warning and so were more surprised.
Forecasting is a tough job but we make it harder than it has to be by committing some familiar cognitive errors. So what are the lessons that we should learn?
First, wishful thinking has struck again. After the Brexit vote, I described the research of the economist Guy Mayraz. At Oxford university’s Centre for Experimental Social Science, Mayraz ran experiments in which participants were told that they were either “farmers”, who would be paid more if wheat prices were high, or “bakers”, who would be paid more if wheat prices were low. They were then shown a graph, purportedly tracking the wheat price, and invited to forecast the future price, with a cash reward for accurate forecasts. Despite the fact that they were being paid for accuracy, the farmer-participants systematically forecast higher wheat prices than the bakers. Everyone predicted what they hoped would happen. Does that sound familiar?
The second lesson is that — as a large experiment conducted by the Good Judgment Project has shown — self-critical, open-minded forecasters do a better job than narrow-minded, overconfident ones. Of course that is obvious — except that open-mindedness is a quality in short supply. Dwelling on our own fallibility is like dwelling on our own mortality: most people find it uncomfortable, so we don’t do it.
Whether you’re sinking a beer with friends or prognosticating on cable news, the social pressure is to make an interesting, confident statement rather than hum and haw about all the ways in which you might be wrong. Confident, eye-catching forecasts are the snack food of analysis and commentary: everybody knows they’re doing us no good but we can’t seem to resist.
It was interesting to see the self-critical dynamic in action at Nate Silver’s political forecasting website FiveThirtyEight. I interviewed Silver at a public event when he was riding high after successful forecasts in 2012. The audience questions were fawning but I was impressed at how Silver kept emphasising that he’d been lucky and future forecasts would be harder.
This time round, FiveThirtyEight botched the analysis of the Republican nomination: the polls said Trump was clearly ahead but nobody could quite believe that. Yet the early failure provoked some introspection. FiveThirtyEight learnt a lesson. While other forecasters were writing off Trump in the presidential race, FiveThirtyEight kept looking at the polls, and continued to declare that it was close.
A third lesson is that we have to keep an open mind that more than one outcome is possible. Too many people equated “Clinton is the favourite” with “Clinton will win”. That’s an obvious error, but it’s common. Even expert forecasters often treat a strong possibility as though it is a certainty. This tendency is one reason that dart-throwing chimps give the experts a run for their money. The chimps make lots of forecasting errors too, but at least they don’t systematically overrate their chances.
Perhaps the best way to keep more than one outcome in mind is to develop scenarios. I’ve written before about the scenario-planning method: scenarios are persuasive, coherent stories about the future, and like all good stories they have a tendency to stick with the listener. A scenario-planner will create at least two contradictory stories — this helps people to keep an open mind, and to prepare for more than one outcome.
FiveThirtyEight does not quite embrace scenario-planning but it did produce a clear account of the kind of polling error that would be necessary to deliver a Trump win — along with a reminder that such polling errors are ubiquitous. Clinton was the favourite even for FiveThirtyEight, but the site presented its readers with a very clear description of how the less-likely prospect of a Trump win would unfold, if it were to happen.
Scenario-thinking is not really intended to produce better forecasts — although I think it does no harm. What it should deliver is preparedness. Trump’s victory has caught a lot of people napping; that’s strange. We take precautionary measures against far less likely events than the victory of a presidential underdog.
Amid many depressing features of the politics of 2016 has been our failure to prepare for perfectly foreseeable possibilities. If there is a silver lining, it’s this: the uncertainties are not going away, so it’s not too late to learn.
Written for and first published in the Financial Times.