Tim Harford The Undercover Economist

Articles published in June, 2019

The dying art of compromise

I don’t often find myself agreeing with Esther McVey, but I wondered this week whether the candidate for leader of the UK Conservative party might accidentally have spoken the truth: “People saying we need a Brexit policy to bring people together are misreading the situation. That is clearly not possible.”

The British do indeed seem in no mood to compromise. The results of elections to the European Parliament produced a thunderous endorsement of parties that proudly reject an attempt to find common ground on Brexit. The Conservatives and Labour, each caught in an awkward straddle, were slaughtered. Labour offered the slogan “let’s bring our country together”. Ha! Voters preferred the Liberal Democrats (“Bollocks to Brexit”) and the Brexit party (“they’re absolutely terrified of us”).

Sometimes an extreme position is the correct one. When King Solomon proposed cutting the baby in half, it wasn’t because he was looking for the middle ground.

Yet a capacity to find compromises is a good thing to have. Positions may differ, but whether we live in the same home or on the opposite side of the planet, we benefit when we can find a way to get along.

If this new distaste for compromise is a problem, it is not the UK’s alone. Positions seem to be hardening everywhere, the sclerotic arteries that may lead to a heart attack for western democracies. Perhaps this is driven by personalities. For a man whose name adorns a book titled The Art of The Deal, Donald Trump is curiously uninterested in negotiating lasting agreements with anyone. Or maybe it is a function of an information ecosystem in which outrage sells.

Perhaps the problems themselves are more intractable. Some issues do not lend themselves to compromise. Brexit is one. Splitting the difference between Remainers and hard Brexiters is less like cutting a cake and more like splattering its ingredients everywhere. Egg on my face, flour on yours, and nobody even partially satisfied.

Abortion is another. There is a principled case to be made for a woman’s absolute right to control her body. There is also a principled case to be made for the absolute right to life of a foetus. But like the unstoppable cannonball and the immovable post, both rights cannot be absolute simultaneously.

In contrast, other complex and emotive problems may still allow for compromise. On climate change, we can shrug and do nothing, or we can turn our economic system upside down, but there is plenty of middle ground between those options. In a trade negotiation, a mutually advantageous outcome is almost always there to be discovered.

Roger Fisher and William Ury’s classic negotiation handbook Getting to Yes advises: focus on the problem rather than the personalities; explore underlying interests rather than explicit positions; and consider options that may open up scope for mutual benefit. We may find a much better way to split the cake if we discover that you scrape the icing into the bin, while I would happily eat it with a spoon. It is sometimes astonishing how far a principled negotiation can go towards giving both sides what they want.

It is clear that we British have failed to follow this advice. Our debate is driven by a bitter focus on personalities, from Theresa May to Nigel Farage to Jeremy Corbyn to the generic “Remoaner elite”. Each side knows what the other wants but has shown very little interest in why they want it. Without sincerely exploring the underlying aims and values of warring tribes there is no chance of finding an outcome everyone can accept.

The US debate also seems the antithesis of Fisher and Ury’s advice. Too many politically active people seek the humiliation of the other tribe. Dismissing compromise as craven appeasement seems to be a winning tactic, particularly in the primary elections that set the tone of US politics.

Compromise, however, is often possible even in unpromising situations. On abortion, for example, it emerges with a focus not on absolute rights but on practicalities. Many people can get behind policies to minimise unwanted pregnancies, and to make abortions safe and regulated rather than dangerous and illicit. It is a middle ground that many countries manage to find.

One can see politics as a competitive sport or a search for solutions. There’s truth in both views. However, a democratic election is far closer to a competition than to a principled negotiation. Do we not wish to see the opposite team soundly thrashed? Do we not boo their villainous antics and laugh at their mishaps? Who wants to play out a nil-nil draw?

I would not want to venerate compromise as the supreme good in politics. Sometimes it really is true that you and I, dear reader, are absolutely right and they are absolutely wrong. (It may even be true that we are absolutely wrong and they are absolutely right.) Either way, the merits of the case must be weighed against the merits of trying to respect everyone. It feels good to win, but this isn’t a fairytale: the losers won’t stamp their feet and vanish through the floor. They — or we — aren’t going anywhere.


Written for and first published in the Financial Times on 31 May 2019.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.

Free email updates

(You can unsubscribe at any time)

Why brilliant people lose their touch

It hasn’t been a great couple of years for Neil Woodford — and it has been just as miserable for the people who have entrusted money to his investment funds. Mr Woodford was probably the most celebrated stockpicker in the UK, but recently his funds have been languishing. Piling on the woes, Morningstar, a rating agency, downgraded his flagship fund this week. What has happened to the darling of the investment community?

Mr Woodford isn’t the only star to fade. Fund manager Anthony Bolton is an obvious parallel. He enjoyed almost three decades of superb performance, retired, then returned to blemish his record with a few miserable years investing in China.

The story of triumph followed by disappointment is not limited to investment. Think of Arsène Wenger, for a few years the most brilliant manager in football, and then an eternal runner-up. Or all the bands who have struggled with “difficult second-album syndrome”.

There is even a legend that athletes who appear on the cover of Sports Illustrated are doomed to suffer the “SI jinx”. The rise to the top is followed by the fall from grace.

There are three broad explanations for these tragic career arcs. Our instinct is to blame the individual. We assume that Mr Woodford lost his touch and that Mr Wenger stopped learning. That is possible. Successful people can become overconfident, or isolated from feedback, or lazy.

But an alternative possibility is that the world changed. Mr Wenger’s emphasis on diet, data and the global transfer market was once unusual, but when his rivals noticed and began to follow suit, his edge disappeared. In the investment world — and indeed, the business world more broadly — good ideas don’t work forever because the competition catches on.

The third explanation is the least satisfying: that luck was at play. This seems implausible at first glance. Could luck alone have brought Mr Wenger three Premier League titles? Or that Mr Bolton was simply lucky for 28 years? Do we really live in such an impossibly random universe?

Perhaps we do. Michael Blastland’s recent book, The Hidden Half, argues that much of the variation we see in the world around us is essentially mysterious. Mr Blastland’s opening example is the marmorkrebs, a kind of crayfish that reproduces parthenogenetically — that is, marmorkrebs lay eggs without mating and those eggs develop into clones of their mothers.

Place two clones into two identical fish tanks and feed them identical food. These genetically identical creatures raised in apparently identical environments produce genetically identical offspring who nevertheless vary dramatically in their size, form, lifespan, fecundity, and behaviour. Sometimes things turn out very differently for no reason that we can discern. We might as well call that reason “luck” as anything else.

This is not to say that skill doesn’t matter — merely that in a competition in which all the leaders are highly skilled, randomness may explain the difference between triumph and failure. Good luck plus skill beats bad luck plus skill any time.

It is easy to underestimate how much chance is at play all around us. The psychologist Daniel Kahneman has recently been studying what he calls “noise”: the variability of judgments for no obvious reason.

A wine expert blind-tasting two glasses from the same bottle will often rate them differently. Pathologists disagree with each other in their judgments of the same biopsy. More disconcertingly, they also disagree with their own prior judgments of the case.

We rarely appreciate just how much inconsistency there is in the judgments we and others make, argues Prof Kahneman. It can hardly be a surprise, then, if past performance is no guarantee of future success.

We should remember, too, that people often achieve outsized success by taking risks or being contrarian. When John Kay examined the forecasting record of economists in the 1990s, he noted that Patrick Minford, an idiosyncratic forecaster, would often produce the best forecast one year and the worst forecast the next. If the consensus is wrong, being an outlier gives you a high chance both of dramatic success and spectacular failure.

We perceive all this randomness through a particular filter, too. Few people make the cover of Sports Illustrated after a run of mediocre luck. They appear after things have been going well, and if the good luck fails to hold then it seems like the SI jinx. More likely it is “regression to the mean”, or in simple terms, a return to business as usual.

We begin paying attention only when someone is producing a remarkable performance. Genius followed by mediocrity is a story arc we all notice. Mediocrity followed by genius just looks like genius — assuming the mediocre performer gets a second chance. Not all do.

So I wish Mr Woodford well. Perhaps he has lost his touch, perhaps the world has changed, or perhaps he has simply been unlucky. It would be nice to know which, but in such matters the world does not always satisfy our curiosity.

Written for and first published in the Financial Times on 24 May 2019.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.

Free email updates

(You can unsubscribe at any time)

The Doris Day effect – when obstacles help us

She had hoped to become a ballet dancer. After her leg was shattered in an accident at the age of 15, she took singing lessons instead. It was a striking detail in the obituaries. If not for that painful setback, the star that was Doris Day would never have risen.

Was the car accident that redirected her career an extraordinary twist in the story of an extraordinary life? Or was it typical of some broader truth about life, that frustrations can actually help us? Perhaps it is true that what does not kill us makes us stronger. It may, in contrast, be that what does not kill us nevertheless slows us down.

The conventional wisdom is that initial advantages tend to snowball into an avalanche of privilege. Sometimes this reflects genuine achievements: a bit of luck with an early teacher sharpens a student’s skills, lifting her into a higher set, which in turn gets her into a better university, then a job with more stimulating peers, and so on.

An egregious example, made famous by Malcolm Gladwell’s book Outliers, is the tendency of elite athletes to be born early in their school year. Being a few months older at the age of five means you are stronger and faster, are more likely to be picked for school teams, get more practice and are still reaping the benefits as an adult athlete. The effect is particularly well-studied among boys playing ice hockey in Canada, and football in a variety of countries.

At other times, well-deserved acclaim is followed by unearned praise. In academia this tendency was named by the sociologist Robert K Merton as “the Matthew Effect” in reference to a biblical verse: “For to every one who has will more be given, and he will have abundance; but from him who has not, even what he has will be taken away.”

If three researchers collaborate on a problem, and one of them already has a Nobel Prize, the laureate tends to earn disproportionate recognition for the joint work. When a teacher and a student work together, the senior researcher is cited because that name is already recognised. The junior is easily forgotten.

In the wider workplace, we have evidence that the luck of graduating in a benign economic climate can lead to a lasting advantage. One researcher, Paul Oyer, found that young PhD and MBA students who started off in favourable job markets were employed in better places with smarter colleagues, and were still doing better a decade later than those who started out in tougher times.

Hannes Schwandt and Till Marco Von Wachter studied the other end of the US labour market to find the story is even worse there: entering the job market during a recession damages anyone’s prospects, but the harm is deeper and lasts longer for less-educated and otherwise disadvantaged groups.

All this suggests that setbacks are setbacks: they drag us down, perhaps disproportionately. Doris Day was an exception, not the rule.

Yet a striking new study suggests that the Doris Day effect is quite real in one particular group of people: young scientists applying for research grants. Yang Wang, Benjamin Jones and Dashun Wang looked at scientists applying for funding from the US National Institutes for Health, with grants averaging $1.3m. In particular, they focused on borderline decisions, comparing those who scraped through to get a grant with those who just missed out. The near-winners and the near-losers were otherwise indistinguishable before the decision point, but afterwards it was the losers who prospered, publishing substantially more highly cited research papers.

We should remember that anyone in a position to nearly secure a million-dollar research grant has presumably enjoyed a few successes along the way at school and university. Failure at this hurdle may be described as an “early career setback”, but it is not comparable to the setback suffered by an undernourished two-year-old with no books in her bedroom.

Still: this is a counterintuitive finding. Yet I was not entirely surprised to encounter it. It may be that many people respond to a setback by bouncing back with renewed determination. It may also be that the failure provokes a rethink and a fresh course of action. Doris Day, after all, did not respond to a shattered leg by trying even harder to become a dancer. She changed her goals and prospered as a result.

We don’t have to be promising young scientists — or aspiring starlets — to benefit from having obstacles placed in our way. Something as mundane as a strike disrupting regular commuting has been shown to push people towards new habits. Three economists who studied data from London’s public transport network found that after a 48-hour strike in London in February 2014, thousands of commuters changed routes and never switched back. They discovered that they’d been doing the commute wrong their entire working lives.

Often failure is simply failure, and a setback is exactly what it seems. But sometimes the obstacle that has been placed in our path might provoke us to look around, and perhaps to discover that a better route was there all along.

Written for and first published in the Financial Times on 17 May 2019.

My book Messy explores some of these ideas in more depth – feel free to order online or through your local bookshop.

Free email updates

(You can unsubscribe at any time)

Why we should favour second guesses over first instincts…

Tension is rising in the Harford household as exams approach and we try to persuade Miss Harford Sr to relax, and Miss Harford Jr to be slightly less relaxed. I’m sure many readers have vivid memories of the exam room, recent or otherwise. But here’s a question about exam technique that suggests a much wider lesson. In a multiple-choice test, you sometimes write down an answer and then have second thoughts. Is it wise to stay with your first instincts, or better to switch?

Most people would advise that the initial answer is usually better than the doubt-plagued second guess. Three-quarters of students think so, according to various surveys over the years. College instructors think so too, by a majority of 55 to 16 per cent. The 2000 edition of Barron’s How to Prepare for the GRE Test is very clear that students should be wary of switching. “Experience indicates that many students who change answers change to the wrong answer.”

This confidence would be reassuring, were it not utterly erroneous. Researchers have been studying this question since the 1920s. They have overwhelmingly concluded both that individual answer changes are more likely to be from wrong to right, and that students who change their answers tend to improve their scores. This gap between perception and reality is stark enough to have earned a name: the “first-instinct fallacy”. No doubt our first instincts are often right, but when we start to have second thoughts, the second thoughts are usually occurring for a reason. It is better to switch. So why don’t we?

Justin Kruger, a psychologist at New York University, has been studying this question. (Prof Kruger is more famous as co-discoverer of the Dunning-Kruger effect: people who are incompetent are too incompetent to realise how incompetent they are.) With his colleagues Derrick Wirtz and Dale Miller he replicated the longstanding findings that college students believe you should trust your first answer in a multiple choice question, and yet that switching to a second answer tends to improve your grades. Then the trio started to explore why.

In one study they showed subjects video based on the TV show Who Wants To Be A Millionaire? and asked them to imagine that they were watching a teammate play, accumulating cash for the team whenever he or she gave the correct answer. Some subjects were shown teammates who always switched on 50/50 questions, while others were shown teammates who always stuck with their first instinct. In the study, both strategies produced identical results, yet subjects watching a switching teammate were more frustrated and critical and had a good memory for the errors.

Another study by Prof Kruger and his colleagues showed that we also have a warped recollection of our own errors in multiple choice tests. We have a rosy memory of sticking to our first instincts, forgetting the failures and exaggerating the successes. We vividly recall switching to the wrong answer and overestimate how often we did so. In short, we remember sticking as having been the best tactic, when in fact switching was better. No wonder the Barron’s guide remarks that “experience” tells us switching is a bad idea. Our own experiences do indeed tell us that, but only because we misremember the lessons of previous switches.

If you — or a loved one — are about to enter exam season, perhaps this evidence-based strategy will be of use. But it’s hard not to see a broader lesson. How often in life do we make a choice and then stick to it despite mounting doubts?

In politics, such questions are aggravated by questions of partisanship and pride. Nobody wants to admit that they were wrong in the face of jeers from those on the opposite side of the political fence. The U-turn is one of the greatest sins in politics, if only because it is so easy to criticise. Either you were wrong before or you are wrong now.

But even in everyday life, we find ourselves clinging to bad choices. Steven Levitt, the co-author of Freakonomics, once conducted a study in which people hesitating over big choices — to leave a spouse, to adopt a child, to quit a job, to start a business — agreed to be guided by a coin toss. Those who had been nudged to act ended up being happier several months on than those who had been nudged to stick with the status quo.

We are prone to cling tightly to the devil we know. The likely explanation is that we are seeking to minimise regret. Forget the old cliché about regretting the things you didn’t do more than the things you did: we regret misfiring action more than misfiring inaction. We starkly remember the times we changed things for the worse, and we more easily forget the times when we failed to change things for the better. It’s not that impulsive action is always the best option, any more than your first answer on a test is always wrong. Instead, the lesson is that if you are hesitating over whether to leave things as they are, you probably needed to make a change some time ago.

Written for and first published in the Financial Times on 10 May 2019.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.

Free email updates

(You can unsubscribe at any time)