Tim Harford The Undercover Economist

Articles published in December, 2018

Why good forecasters become better people

So, what’s going to happen next, eh? Hard to say: the future has a lotta ins, a lotta outs, a lotta what-have-yous.

Perhaps I should be more willing to make bold forecasts. I see my peers forecasting all kinds of things with a confidence that only seems to add to their credibility. Bad forecasts are usually forgotten and you can milk a spectacular success for years.

Yet forecasts are the junk food of political and economic analysis: tasty to consume but neither satisfying nor healthy in the long run. So why should they be any more wholesome to produce?

The answer, it seems, is that those who habitually make forecasts may turn into better people. That is the conclusion suggested by a research paper from three psychologists, Barbara Mellers, Philip Tetlock and Hal Arkes.

Prof Tetlock won attention for his 2005 book Expert Political Judgment (UK) (US), which used the simple method of asking a few hundred experts to make specific, time-limited forecasts such as “Will Italy’s government debt/GDP ratio be between 70 and 90 per cent in December 1998?” or “Will Saddam Hussein be the president of Iraq on Dec 31 2002?”

It is only a modest oversimplification to summarise Prof Tetlock’s results using the late William Goldman’s aphorism: nobody knows anything.

Yet Profs Mellers, Tetlock and Don Moore then ran a larger forecasting tournament and discovered that a small number of people seem to be able to forecast better than the rest of us. These so-called superforecasters are not necessarily subject-matter experts, but they tend to be proactively open-minded, always looking for contrary evidence or opinions.

There are certain mental virtues, then, that make people better forecasters. The new research turns the question around: might trying to become a better forecaster strengthen such mental virtues? In particular, might it make us less polarised in our political views?

Of course there is nothing particularly virtuous about many of the forecasts we make, which are often pure bluff, attention-seeking or cheerleading. “We are going to make America so great again” (Donald Trump, February 2016); “There will be no downside to Brexit, only a considerable upside” ( David Davis, October 2016); “If this exit poll is right . . . I will publicly eat my hat” (Paddy Ashdown, May 2015). These may all be statements about the future, but it seems reasonable to say that they were never really intended as forecasts.

A forecasting tournament, on the other hand, rewards a good-faith effort at getting the answer right. A serious forecaster will soon be confronted by the gaps in his or her knowledge. In 2002, psychologists Leonid Rozenblit and Frank Keil coined the phrase “the illusion of explanatory depth”. If you ask people to explain how a flush lavatory actually works (or a helicopter, or a sewing machine) they will quickly find it is hard to explain beyond hand-waving. Most parents discover this when faced by questions from curious children.

Yet subsequent work has shown that asking people to explain how the US Affordable Care Act or the European Single Market work prompts some humility and, with it, political moderation. It seems plausible that thoughtful forecasting has a similar effect.

Good forecasters are obliged to consider different scenarios. Few prospects in a forecasting tournament are certainties. A forecaster may believe that parliament is likely to reject the deal the UK has negotiated with the EU, but he or she must seriously evaluate the alternative. Under which circumstances might parliament accept the deal instead? Again, pondering alternative scenarios and viewpoints has been shown to reduce our natural overconfidence.

My own experience with scenario planning — a very different type of futurology than a forecasting tournament — suggests another benefit of exploring the future. If the issue at hand is contentious, it can feel safer and less confrontational to talk about future possibilities than to argue about the present.

It may not be so surprising, then, that Profs Mellers, Tetlock and Arkes found that forecasting reduces political polarisation. They recruited people to participate in a multi-month forecasting tournament, then randomly assigned some to the tournament and some to a non-forecasting control group. (A sample question: “Will President Trump announce that the US will pull out of the Trans-Pacific Partnership during the first 100 days of his administration?”)

At the end of the experiment, the forecasters had moderated their views on a variety of policy domains. They also tempered their inclination to presume the opposite side was packed with extremists. Forecasting, it seems, is an antidote to political tribalism.

Of course, centrism is not always a virtue and, if forecasting tournaments are a cure for tribalism, then they are a course of treatment that lasts months. Yet the research is a reminder that not all forecasters are blowhards and bluffers.

Thinking seriously about the future requires keeping an open mind, understanding what you don’t know, and seeing things as others see them. If the end result is a good forecast, perhaps we should see that as the icing on the cake.


Written for and first published in the Financial Times on 23 Nov 2018.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.

Free email updates

(You can unsubscribe at any time)

The economist’s guide to the perfect Christmas

It was snowing even in London this week. Surely now it’s time to get serious about Christmas — and who better to give the perfect yuletide advice than an economist? (I also plan to include the best ideas from psychology, and call them “behavioural economics”; this is a proven formula.)

Economists are much needed at this time of year since Christmas is, more than anything, a consumerist blowout. It has been for well over a century. Joel Waldfogel, economist and author of Scroogenomics (UK) (US), comments that “just as every generation imagines that it invented sex, every generation imagines that it invented the vulgar commercialisation of Christmas”. Prof Waldfogel has tracked the size of the spending boom in the US in December, compared with November and January. It has been sizeable at least since the 1930s and probably much longer than that. If anything, Christmas stands out less in the spending data than it did three generations ago.

As an economist, I have nothing against this rampant consumerism — but I do wonder whether there is a way to enjoy a better Christmas.

Here is my three-point plan.

Step one: beware the efficient presents hypothesis. This is a variant on the efficient markets hypothesis, which says (roughly) that there are no bargains to be had on the stock market because they’ve already been noticed and snapped up. Similarly, the efficient presents hypothesis says that all the most suitable gifts have already been purchased — typically by recipients who have decided to treat themselves.

I have already fallen foul of the efficient presents hypothesis this year; carefully selecting a pair of extra-warm socks in precisely the style and size my wife prefers, I was dismayed when a parcel containing a duplicate pair arrived at our house a few days later. She was one step ahead of me in picking her own presents. The efficient presents hypothesis is not always true, any more than the regular efficient markets hypothesis. It is nevertheless true often enough to take seriously.

Step two: adopt a passive gift-buyer strategy. Again, the parallel with investment should be clear. You can achieve excellent investment results simply by making regular payments into passive index tracker funds. This strategy is dull and unimaginative, leaving no room for flair or good judgment. Nevertheless it works — partly because good judgment is scarce and flair often counterproductive. Active managers are often unable to outperform the stock market by enough to justify their fees. Individual investors tend to trade too often, buy high and sell low. Regular passive investment may be boring but it avoids these traps.

The ultimate passive gift is cash. Just like tracker funds it is utterly unimaginative yet a surprisingly difficult benchmark to beat. Many active gift-buyers swear they can get more than £50-worth of joy out of a £50 present, but Prof Waldfogel has good evidence that most of them fall well short. Gift-givers, like stock pickers, tend to overrate their abilities. (At least gift-givers have an excuse: they receive no feedback. Nobody is going to tell you that they hate the present you bought for them, but if your stock portfolio crashes it is hard not to notice.)

Since giving cash is often socially unacceptable, there is another passive approach that works well: find a wishlist, or just ask the recipient what they would like. Just as passive investment in index funds robs the stockpicking game of its daring and mystique, simply consulting a list seems robotic and joyless. But — as Francis Flynn of Stanford and Francesca Gino of Harvard have found — it is rarely perceived that way by recipients. While gift-givers hesitate to fall back on a wish list, recipients prefer items they have indicated that they actually want. They still think of the present purchaser as perfectly thoughtful: after all, someone took the trouble to find out what you wanted.

Step three: give the gift of time and attention. With all the effort you’ve saved ordering gifts from wish lists or simply writing cheques, see friends and enjoy the rituals of Christmas. Fresh from her wishlist research, Prof Gino has been part of a team studying the way family rituals influence our experience of seasonal festivities. Whether the rituals are secular or sacred, they are correlated with liking your family, feeling more satisfied with life and paying closer attention to your experiences. Exactly what causes what is not clear, but the idea that a good Christmas tradition brings people together is a sensible one. Too often, we lack the time because we are spending countless hours running around shops buying things that nobody will ever tell you they hated, already owned or both.

One could do worse than the reformed Ebenezer Scrooge, who, Dickens tells us, “knew how to keep Christmas well, if any man alive possessed the knowledge”. On Christmas morning the only physical gift he gives is a prize turkey, having been assured on good ghostly authority that it is much needed. Other than that, he gives time and money, notably a pay rise for Bob Cratchit.

Money! That’s the Christmas spirit. God bless us, every one!

Written for and first published in the Financial Times on 15 December 2017.

Free email updates

(You can unsubscribe at any time)

Brexit, Trump, and how politics loses the capacity to shock

How often do I find myself utterly unsurprised by a news headline that should be shocking? Whether it’s Donald Trump declaring the media to be “the true Enemy of the People” after bombs had been sent to CNN offices, or the UK government planning to charter a flotilla to keep the nation supplied with broccoli and penicillin in a no-deal Brexit scenario, I merely shrug. Of course it’s appalling, I think, but it’s the logical continuation of what has been said and done already.

So let’s talk about the psychologist Stanley Milgram. Milgram is most notorious for his electric shock experiments in the 1960s. He recruited unsuspecting members of the public to participate in a “study of memory”. On showing up at the laboratory, they drew lots with another participant to see who would be “teacher” and who “learner”. Once the learner was strapped into an electric chair, the teacher retreated into another room to take control of a shock machine.

As the learner failed to answer questions correctly, the teacher was asked to administer steadily increasing electric shocks. Many proved willing to deliver possibly fatal shocks — despite having received a painful shock themselves as a demonstration, despite the learner having already complained of a heart condition, despite the screams of pain and the pleadings to be released from the other side of the wall, and despite the fact that the switches on the shock machine read “Danger: Severe Shock, XXX”.

Of course, there were no shocks — the man screaming from the nearby room was pretending. Yet the research exerts a horrifying fascination. In the best known study, 65 per cent of experimental subjects went all the way to 450 volts, applying shocks long after the man in the other room had fallen silent.

In the shadow of the Holocaust, which influenced Milgram’s research agenda, the obvious conclusion was that we will do terrible things if an authority figure requires them. But psychologists no longer draw that lesson from Milgram’s experiment.

Behind The Shock Machine (UK) (US) a history by Gina Perry, reminds us that Milgram’s experimental set-ups varied. In most of them, more than half of participants refused to continue. And Alex Haslam, a psychologist who has re-examined the studies, found that direct orders backfired. When people complied it was not because they were ordered, but because they were persuaded.

One often overlooked detail is that Milgram’s shock machine had 30 settings in increments of 15 volts. It’s hard to object to giving someone a tiny 15-volt shock. And if you’ve decided that 15 volts is fine, then why draw the line at 30 volts? Why draw the line at 45? Why draw the line anywhere?

At 150 volts, the “learner” yelled out in distress. Some people stopped at that point. But those who continued past 150 volts were overwhelmingly likely then to persist to the full 450 volts. They were in too deep. Refusing to administer a shock of 225 volts would be an implicit admission that they had been wrong to deliver 210.

Perhaps we need to turn to another great mid-century psychologist, Leon Festinger, for an explanation. Festinger is best known for the theory of “cognitive dissonance”, the discomfort of holding two contradictory notions — such as “I’m a decent person” and “I just hit that poor guy with 210 volts”.

Festinger demonstrated that we are able to summon up considerable reserves of wishful thinking and selective memory in order to restore consistency. The further people slid into the Milgram experiment, the harder they worked to convince themselves that it was all in a good cause, or that no real harm was being done, or both.

Seeing the experiment described in textbooks half a century later, it still seems perplexing. And perhaps future students of history will be baffled to see recent events concisely summarised. The Republicans, party of “family values”, confirmed a Supreme Court justice nominee after he was accused of sexual assault? Could they really not find someone better?

But these future students will not see the 15-volt increments that got us to this destination. Electing a president who has boasted of his own sexual depredations meant crossing the 150-volt line. Once you’ve found a way to laugh off the issue, it is hard to treat it with gravity thereafter.

Although Brexit is a very different business, it, too, will make little sense to future generations unless they see that we got there 15 volts at a time. It turns out the single market requires free movement of labour? Zap! We’ve discovered that the border with Ireland is a sensitive issue? Zap! The funding of the Leave.EU campaign is being investigated by the National Crime Agency? Zap!

The consolation is that democracies provide us with moments in which we can step back and think about the direction we are taking. The recent US midterm elections were one; there will be others. “I have a choice,” responded one Milgram subject, when ordered to increase the voltage. “I’m not going to go ahead with it.” That is worth remembering.



Written for and first published in the Financial Times on 16 November 2018.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.

Free email updates

(You can unsubscribe at any time)

Messy desks and benign neglect allow ideas to grow

My daughter is about to receive a new desk, so in order to clear space for it we were obliged to hack our way through the undergrowth of a 12-year-old’s bedroom. We found a half-assembled jigsaw puzzle from last Christmas; three separate sets of worn pyjamas scrunched up and stored in diverse locations; and empty sweet wrappers from Halloween. More alarmingly, there were empty sweet wrappers from Easter.

I am trying my best to treat with equanimity the discovery of a novel ecosystem under my roof. This is because I have come to believe that many spaces work a great deal better if subjected to a sustained period of benign neglect.

Consider the office cubicle. Some people pile their desks with everything from old newspapers to unwashed mugs; others are fastidiously tidy. (I fluctuate.) I’m not saying that people with messy desks are more productive, although there’s some evidence that they are; I’m just saying that if your colleague is a messy-desker then he or she should be allowed to get on with it.

Support for this position comes from a study conducted by two psychologists, Alex Haslam and Craig Knight. A few years ago they set up simple office spaces in which they asked experimental subjects to spend an hour doing administrative tasks. Messrs Haslam and Knight wanted to understand what made people productive and happy, and they tested four arrangements in a randomised trial. One was minimalist: chair, desk, bare walls. A second was softened with tasteful prints and some greenery. Workers were happier there, and got more done.

The kicker comes with the third and fourth arrangements. In each case, workers were invited to rearrange the pictures and pot-plants as they wished before settling down to work. But while some were then left to their labours, others were second-guessed by an experimenter who stepped in and found a pretext to rearrange everything.

This, unsurprisingly, drove people mad. “I wanted to hit you,” one participant later admitted. Empowering people to lay out their own space led to happier, more productive workers. Stripped of that freedom, everyone’s productivity fell and some felt quite ill.

The principle of benign neglect may well operate on a larger scale. Consider Building 20, one of the most celebrated structures at Massachusetts Institute of Technology. The product of wartime urgency, it was designed one afternoon in the spring of 1943, then hurriedly assembled out of plywood, breeze-blocks and asbestos. Fire regulations were waived in exchange for a promise that it would be pulled down within six months of the war’s end; in fact the building endured, dusty and uncomfortable, until 1998.

During that time, it played host not only to the radar researchers of Rad Lab (nine of whom won Nobel Prizes) but one of the first atomic clocks, one of the first particle accelerators, and one of the first anechoic chambers — possibly the one in which composer John Cage conceived 4’33. Noam Chomsky revolutionised linguistics there. Harold Edgerton took his high-speed photographs of bullets hitting apples. The Bose Corporation emerged from Building 20; so did computing powerhouse DEC; so did the hacker movement, via the Tech Model Railroad Club.

Building 20 was a success because it was cheap, ugly and confusing. Researchers and departments with status would be placed in sparkling new buildings or grand old ones — places where people would protest if you nailed something to a door. In Building 20, all the grimy start-ups were thrown in to jostle each other, and they didn’t think twice about nailing something to a door — or, for that matter, for taking out a couple of floors, as Jerrold Zacharias did when installing the atomic clock.

As Stewart Brand drily remarked in How Buildings Learn (UK) (US) Building 20 worked because “nobody cares what you do in there”.

If benign neglect works for your colleague’s desk and it works for an entire building, what about a grander scale still? What about a city neighbourhood? Up to a point, yes: even cities benefit from being left alone in certain ways. Of course, potholes must be fixed, bins emptied and charging points for electric vehicles installed. But Jane Jacobs argued in The Death And Life of Great American Cities (UK) (US) that cities desperately need old buildings, and not just glorious masterpieces but “a good lot of plain, ordinary, low-value old buildings, including some rundown old buildings”.

Her reasoning: cities are always in need of new experiments and economically marginal activities. “Neighbourhood bars . . . good bookshops . . . studios, galleries . . . hundreds of ordinary enterprises” all need somewhere cheap.

There’s nothing wrong with new buildings, argued Ms Jacobs, frustratingly for those who hold her up as a Nimby icon. But they should not be built everywhere all at once. Something has to be neglected and run down, or the city has no soil from which new buds can shoot.

There is always a balance to be struck. Every old building was once new. Every desk needs the occasional wipe. And my daughter is currently engaged in an extended programme of supervised room-tidying. Yet neglect is undervalued. Sometimes we need to learn when to leave well alone.


The ideas in this column are more fully expressed in my book “Messy: How To Be Creative and Resilient in a Tidy-Minded World”. It’s available in paperback both in the US and the UK – or through your local bookshop.


Written for and first published in the Financial Times on 9 November 2018.

Free email updates

(You can unsubscribe at any time)


  • 1 Twitter
  • 3 RSS
  • 5 Podcasts
  • 6 Facebook


  • Fifty Inventions That Shaped the Modern Economy
  • Messy
  • The Undercover Economist Strikes Back
  • Adapt
  • Dear Undercover Economist
  • The Logic of Life
  • The Undercover Economist

Free Email Updates

Enter your email address to receive notifications of new articles by email (you can unsubscribe at any time).

Join 188,795 other subscribers.