Tim Harford The Undercover Economist

Articles published in June, 2019

The Doris Day effect – when obstacles help us

She had hoped to become a ballet dancer. After her leg was shattered in an accident at the age of 15, she took singing lessons instead. It was a striking detail in the obituaries. If not for that painful setback, the star that was Doris Day would never have risen.

Was the car accident that redirected her career an extraordinary twist in the story of an extraordinary life? Or was it typical of some broader truth about life, that frustrations can actually help us? Perhaps it is true that what does not kill us makes us stronger. It may, in contrast, be that what does not kill us nevertheless slows us down.

The conventional wisdom is that initial advantages tend to snowball into an avalanche of privilege. Sometimes this reflects genuine achievements: a bit of luck with an early teacher sharpens a student’s skills, lifting her into a higher set, which in turn gets her into a better university, then a job with more stimulating peers, and so on.

An egregious example, made famous by Malcolm Gladwell’s book Outliers, is the tendency of elite athletes to be born early in their school year. Being a few months older at the age of five means you are stronger and faster, are more likely to be picked for school teams, get more practice and are still reaping the benefits as an adult athlete. The effect is particularly well-studied among boys playing ice hockey in Canada, and football in a variety of countries.

At other times, well-deserved acclaim is followed by unearned praise. In academia this tendency was named by the sociologist Robert K Merton as “the Matthew Effect” in reference to a biblical verse: “For to every one who has will more be given, and he will have abundance; but from him who has not, even what he has will be taken away.”

If three researchers collaborate on a problem, and one of them already has a Nobel Prize, the laureate tends to earn disproportionate recognition for the joint work. When a teacher and a student work together, the senior researcher is cited because that name is already recognised. The junior is easily forgotten.

In the wider workplace, we have evidence that the luck of graduating in a benign economic climate can lead to a lasting advantage. One researcher, Paul Oyer, found that young PhD and MBA students who started off in favourable job markets were employed in better places with smarter colleagues, and were still doing better a decade later than those who started out in tougher times.

Hannes Schwandt and Till Marco Von Wachter studied the other end of the US labour market to find the story is even worse there: entering the job market during a recession damages anyone’s prospects, but the harm is deeper and lasts longer for less-educated and otherwise disadvantaged groups.

All this suggests that setbacks are setbacks: they drag us down, perhaps disproportionately. Doris Day was an exception, not the rule.

Yet a striking new study suggests that the Doris Day effect is quite real in one particular group of people: young scientists applying for research grants. Yang Wang, Benjamin Jones and Dashun Wang looked at scientists applying for funding from the US National Institutes for Health, with grants averaging $1.3m. In particular, they focused on borderline decisions, comparing those who scraped through to get a grant with those who just missed out. The near-winners and the near-losers were otherwise indistinguishable before the decision point, but afterwards it was the losers who prospered, publishing substantially more highly cited research papers.

We should remember that anyone in a position to nearly secure a million-dollar research grant has presumably enjoyed a few successes along the way at school and university. Failure at this hurdle may be described as an “early career setback”, but it is not comparable to the setback suffered by an undernourished two-year-old with no books in her bedroom.

Still: this is a counterintuitive finding. Yet I was not entirely surprised to encounter it. It may be that many people respond to a setback by bouncing back with renewed determination. It may also be that the failure provokes a rethink and a fresh course of action. Doris Day, after all, did not respond to a shattered leg by trying even harder to become a dancer. She changed her goals and prospered as a result.

We don’t have to be promising young scientists — or aspiring starlets — to benefit from having obstacles placed in our way. Something as mundane as a strike disrupting regular commuting has been shown to push people towards new habits. Three economists who studied data from London’s public transport network found that after a 48-hour strike in London in February 2014, thousands of commuters changed routes and never switched back. They discovered that they’d been doing the commute wrong their entire working lives.

Often failure is simply failure, and a setback is exactly what it seems. But sometimes the obstacle that has been placed in our path might provoke us to look around, and perhaps to discover that a better route was there all along.

Written for and first published in the Financial Times on 17 May 2019.

My book Messy explores some of these ideas in more depth – feel free to order online or through your local bookshop.

Free email updates

(You can unsubscribe at any time)

Why we should favour second guesses over first instincts…

Tension is rising in the Harford household as exams approach and we try to persuade Miss Harford Sr to relax, and Miss Harford Jr to be slightly less relaxed. I’m sure many readers have vivid memories of the exam room, recent or otherwise. But here’s a question about exam technique that suggests a much wider lesson. In a multiple-choice test, you sometimes write down an answer and then have second thoughts. Is it wise to stay with your first instincts, or better to switch?

Most people would advise that the initial answer is usually better than the doubt-plagued second guess. Three-quarters of students think so, according to various surveys over the years. College instructors think so too, by a majority of 55 to 16 per cent. The 2000 edition of Barron’s How to Prepare for the GRE Test is very clear that students should be wary of switching. “Experience indicates that many students who change answers change to the wrong answer.”

This confidence would be reassuring, were it not utterly erroneous. Researchers have been studying this question since the 1920s. They have overwhelmingly concluded both that individual answer changes are more likely to be from wrong to right, and that students who change their answers tend to improve their scores. This gap between perception and reality is stark enough to have earned a name: the “first-instinct fallacy”. No doubt our first instincts are often right, but when we start to have second thoughts, the second thoughts are usually occurring for a reason. It is better to switch. So why don’t we?

Justin Kruger, a psychologist at New York University, has been studying this question. (Prof Kruger is more famous as co-discoverer of the Dunning-Kruger effect: people who are incompetent are too incompetent to realise how incompetent they are.) With his colleagues Derrick Wirtz and Dale Miller he replicated the longstanding findings that college students believe you should trust your first answer in a multiple choice question, and yet that switching to a second answer tends to improve your grades. Then the trio started to explore why.

In one study they showed subjects video based on the TV show Who Wants To Be A Millionaire? and asked them to imagine that they were watching a teammate play, accumulating cash for the team whenever he or she gave the correct answer. Some subjects were shown teammates who always switched on 50/50 questions, while others were shown teammates who always stuck with their first instinct. In the study, both strategies produced identical results, yet subjects watching a switching teammate were more frustrated and critical and had a good memory for the errors.

Another study by Prof Kruger and his colleagues showed that we also have a warped recollection of our own errors in multiple choice tests. We have a rosy memory of sticking to our first instincts, forgetting the failures and exaggerating the successes. We vividly recall switching to the wrong answer and overestimate how often we did so. In short, we remember sticking as having been the best tactic, when in fact switching was better. No wonder the Barron’s guide remarks that “experience” tells us switching is a bad idea. Our own experiences do indeed tell us that, but only because we misremember the lessons of previous switches.

If you — or a loved one — are about to enter exam season, perhaps this evidence-based strategy will be of use. But it’s hard not to see a broader lesson. How often in life do we make a choice and then stick to it despite mounting doubts?

In politics, such questions are aggravated by questions of partisanship and pride. Nobody wants to admit that they were wrong in the face of jeers from those on the opposite side of the political fence. The U-turn is one of the greatest sins in politics, if only because it is so easy to criticise. Either you were wrong before or you are wrong now.

But even in everyday life, we find ourselves clinging to bad choices. Steven Levitt, the co-author of Freakonomics, once conducted a study in which people hesitating over big choices — to leave a spouse, to adopt a child, to quit a job, to start a business — agreed to be guided by a coin toss. Those who had been nudged to act ended up being happier several months on than those who had been nudged to stick with the status quo.

We are prone to cling tightly to the devil we know. The likely explanation is that we are seeking to minimise regret. Forget the old cliché about regretting the things you didn’t do more than the things you did: we regret misfiring action more than misfiring inaction. We starkly remember the times we changed things for the worse, and we more easily forget the times when we failed to change things for the better. It’s not that impulsive action is always the best option, any more than your first answer on a test is always wrong. Instead, the lesson is that if you are hesitating over whether to leave things as they are, you probably needed to make a change some time ago.

Written for and first published in the Financial Times on 10 May 2019.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.

Free email updates

(You can unsubscribe at any time)


  • 1 Twitter
  • 2 Flickr
  • 3 RSS
  • 4 YouTube
  • 5 Podcasts
  • 6 Facebook


  • Fifty Inventions That Shaped the Modern Economy
  • Messy
  • The Undercover Economist Strikes Back
  • Adapt
  • Dear Undercover Economist
  • The Logic of Life
  • The Undercover Economist

Search by Keyword

Tim’s Tweets

Free Email Updates

Enter your email address to receive notifications of new articles by email (you can unsubscribe at any time).

Join 173,006 other subscribers

Do NOT follow this link or you will be banned from the site!