Undercover Economist

US health care is literally killing people

It is astonishing how far the debate on healthcare has moved in the US, at least for the Democrats. Not long ago offering universal, government-funded healthcare was viewed as tantamount to communism; now, it’s a touchstone of many presidential hopefuls.

Not before time. The US healthcare system is a monument to perverse incentives, unintended consequences and political inertia. It is astonishingly bad — indeed, it’s so astonishingly bad that even people who believe it’s bad don’t appreciate quite how bad it is.

I don’t say this out of any great devotion to the UK alternative. The National Health Service works well enough for a vast tax-funded bureaucracy, but it might work better if we didn’t view any attempt at reform as the desecration of a holy institution. Nor do I have bad experiences of US healthcare. My daughter was born in America, where my family had sensitive and expert medical care. But that’s what you’d expect with a good health insurance plan — something that many Americans don’t have.

Around 27m people — 10 per cent of the non-elderly US population — have no insurance at all. That is precarious, given that a serious illness or accident could incur bankruptcy-inducing costs. Yet the astonishingly large number of people living on the edge is still progress: before the passage of the Affordable Care Act under President Barack Obama, the figure was closer to 45m people.

It’s this lack of anything resembling universal access that seems most grotesque to observers from other rich nations. But it’s just the beginning of the costs that the US health system imposes on Americans.

The financial costs are most obvious, and they are truly extraordinary. For a family of four, the US system costs about $13,000 a year more than that of Switzerland, which itself is substantially more expensive than any other. The US system costs more than twice as much, per person, as the universal coverage provided by the UK’s NHS. Even the government-funded part of the US system costs more per capita than the NHS.

Why so expensive? It’s because US doctors prescribe more treatments, and those treatments cost much more than they do elsewhere. Most governments limit the price of treatments, freeriding on the US market to stimulate investment in medicine. American hospitals and drug companies have enormous leeway to raise prices — insurers have limited bargaining power, and uninsured patients even less.

Nor is all this money bringing any obvious reward. Compared with other rich countries, the US ranks at or near the bottom on life expectancy, infant mortality, adolescent pregnancy, sexually transmitted infections, drug-related mortality, obesity, diabetes, heart disease, lung disease and arthritis. No, the healthcare system can’t be blamed for all that — but it is hardly covering itself with glory.

One of the striking tragedies of modern America, brought to light by the research of the economists Anne Case and Angus Deaton, has been the phenomenon of “deaths of despair”, from suicide, alcohol abuse and overdoses. Such deaths go a long way to explaining why mortality rates for middle-aged white Americans have stagnated or perhaps even risen in the US, while falling fast in other rich countries.

I recently had the opportunity to ask Prof Case and Sir Angus to what extent the US healthcare system was to blame. Their answer, in a nutshell: it would be an exaggeration to blame the system entirely but not a gross exaggeration.

The most obvious connection is that the opioids that have played such a role in these deaths of despair were supplied by the healthcare system. Opioids are a simple and profitable palliative for a widespread condition (“I’m in pain”) rather than a cure for anything. Doctors and drug companies made more money if they prescribed more opioids, and human nature being human nature, found ways to justify that decision to themselves.

The dysfunction of the US healthcare system has also eaten away at American wellbeing in other ways. Those extraordinary costs — more than $10,000 per person — must be paid by someone. When they are paid by employers, through workplace health plans, rising healthcare spending becomes a substitute for the rising wages that workers so desperately want.

And those extortionate costs also give employers a powerful reason to jettison staff at every opportunity, employing freelancers and subcontractors in the hope of cutting the cost of employer-sponsored health insurance. As a result, people feel disconnected from the workplace. Jobs become insecure ways to scrape a living, rather than sources of identity and pride. For many, despair follows.

Such problems are easier to diagnose than to cure. Reforming American healthcare will require an almighty effort. With politics gridlocked and soaking in lobbyist money, it’s not obvious that the US government is capable of running the kind of healthcare system that works elsewhere — even if Congress decides to try. But try it must, because the status quo is a tragedy.

Written for and first published in the Financial Times on 12 July 2019.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.

The strange power of the idea of “average”

“While nothing is more uncertain than a single life, nothing is more certain than the average duration of a thousand lives.” The statement is often attributed to the 19th-century mathematician Elizur Wright, who not coincidentally was a life insurance geek. But buried in the aphorism is a humdrum word concealing a powerful idea: the “average”.

The idea of taking an average — that is, of adding up (say) a hundred lifespans and dividing the total by a hundred, to produce the arithmetic mean — seems absurdly simple. But Stephen Stigler, a historian of statistics, reckons it is the most radical statistical operation ever devised. I am inclined to agree. The mean has a strange power over the way we think, and not always a benign one.

We do not know who invented the arithmetic mean. The statistician Churchill Eisenhart once tried to trace its history. It was originally used as a way of combining various observations that should be identical, but were not — for example, estimates of the direction of magnetic north.

In 1635 the mathematician Henry Gellibrand used the word “meane” to describe the midpoint of a lowest and highest number — not the same thing — but by 1668, a person known as “DB” was quoted in the Transactions of the Royal Society describing “taking the mean” of five values casually enough to make it clear that the concept was by then established.

Why is this such a powerful idea? As Prof Stigler puts it in The Seven Pillars of Statistical Wisdom, “you can actually gain information by throwing information away”. This is true in the straightforward sense that too many numbers become confusing: more than 50m people died last year, but if I could somehow show you a hundred-mile long printout of all their ages at death, you might struggle to learn much from it.

But the mean also eliminates errors. In the context that Gellibrand and DB were writing, taking an average cancels out mistakes in the original observations. This was by no means obvious. When confronted with contradictory measurements, the instinct of mathematicians had been to figure out which one was best and to dismiss the rest. But taking the average was a far better way to eliminate error.

And yet in this method lies a trap, because not all variation is error. The trap was sprung in the 1830s by the hugely influential statistician Adolphe Quetelet, who was an astronomer as well as a founder of the idea of “social physics” — using statistical techniques to understand humans and their societies.

Quetelet asked us to imagine that a thousand sculptors had made a copy of a famous statue of a gladiator. Each copy would have some errors or imperfections — but on average, they would be a perfect copy. So far, so good. But then, Quetelet continued, if we measured a thousand real soldiers and averaged their body measurements, we could get the ideal, perfect soldier. “L’homme moyen” — the average man — was Quetelet’s benchmark for perfection. (What about “la femme moyenne”? Well, quite.)

But as Todd Rose points out in The End of Average, Quetelet’s logic isn’t just andro-centric. It’s nonsense. A tall copy of a statue may be an error, but a tall soldier is not — and may well benefit from having superior reach.

Quetelet did not lack for critics. His contemporary, the mathematician and proto-economist AA Cournot, correctly argued that the Average Man probably didn’t exist. Victorian statistician Francis Galton was fascinated by averages, yet asserted: “No statistician dreams of combining objects into the same generic group that do not cluster towards a common centre . . . if we do so the result is monstrous and meaningless.”

But this idea of the average as perfection did not die. In an age of mass production it was too convenient. Production-line managers and modernist architects found it easy to design for the average person.

In 1943, the sculptor Abram Belskie and obstetrician Robert Dickinson turned Quetelet’s metaphor into reality, carving statues of “Normman” and “Norma”, based on the average measurements of 15,000 young adults. The US press loved Norma in particular: she was regarded as female perfection, at least from the male perspective. (It would be a stretch to describe the statue as “monstrous”.) Yet a competition to find an actual woman who matched her dimensions did not succeed. Being precisely average is not the same thing as being perfect — but it is just as rare.

We no longer have to fall into this trap. It is still hard to personalise rather than standardise — but it is not impossible. Drugs are not best evaluated by the average effect over thousands of patients. Social care will fail if it is designed only for the average recipient, or education for the average pupil. A forecasting model that is correct on average may be a very dangerous model indeed.

Elizur Wright was quite correct to declare that a single life is uncertain; but we should never leap to the conclusion that an average life is ideal.

Written for and first published in the Financial Times on 5 July 2019.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.

How economics can raise its game

How can economics become a more insightful discipline? Should it aim to be more like physics, with its precision and predictive power? Or should economists emulate anthropologists or historians, immersing themselves in the details of the particular and the unquantifiable?

There’s a case to be made either way. Some critics argue that economics is missing better physics: it got stuck in the 19th century with fusty old ideas like marginal analysis and equilibrium, and missed out on cool ideas like chaos theory and phase transitions that promise to shed insights on economic complexity or sudden crises. (See, for example, Philip Ball’s excellent book, Critical Mass.)

Others say that economics needs to put the mathematics down and back slowly away. As Immanuel Kant put it, “out of the crooked timber of humanity, no straight thing was ever made”, so economists should be less fond of putting rulers against everything.

The fact that both views have the ring of plausibility suggests that this is a tougher challenge than it might appear from the sidelines. Now a new paper addresses the question from the heart of academic economics: Nobel laureate George Akerlof, writing in the Journal of Economic Literature.

Prof Akerlof, now at Georgetown, argues that the academic discipline of economics rewards “hard” rather than “soft” research with publication in the top journals, and therefore with promotion and status. We know “hard” when we see it: numbers are harder than words, quantities harder than qualities. Causation is harder than correlation. Physics is “hard” and sociology “soft”.

There is much to be said for hard science. The downside is that certain questions cannot be answered — or perhaps even asked — in precise, mathematical, causal terms. They are still important, and if economics insists on “hard” methods it will overlook them.

Another Nobel laureate, the late Gary Becker, reflected on his college studies: “I began to lose interest in economics . . . because it did not seem to deal with important social problems. I contemplated transferring to sociology, but found that subject too difficult.”

What is fascinating about that remark is that Becker earned his prize by applying the hard-ish tools of economics to areas that seemed to belong to softer disciplines: addiction, discrimination, marriage and education. How well he succeeded remains a matter of debate. Personally I believe he contributed a lot.

Yet plenty of sociology remains outside the reach of the economists’ tools: it is important but too soft — or in Becker’s words, “too difficult”. More awkwardly for the economics profession, some key economic questions also seem more likely to yield to soft than hard approaches: what are the obstacles to social mobility? Where does innovation come from? Can we strengthen the institutions that matter for prosperity?

Beyond any particular problem there is also the challenge of combining insights from highly specialised subfields. Raghuram Rajan, when he was chief economist of the IMF, came closest to predicting the 2008 financial crisis. He later observed that economists had written insightfully on all the key issues but had lacked someone capable of putting all the pieces together.

Is this also a hard/soft problem? Prof Akerlof thinks so. He argues that the bias towards hard analysis also produces a bias towards specialised silo thinking, and that being a generalist is something of a soft skill.

I’m not sure that’s right. Certain mathematical tools are both highly portable and distinctly hard-edged. But it is surely true that the kind of synthesis that would have identified the looming crisis would have been too discursive to be published in a top economic journal.

So I am not completely persuaded that economics is, on average, “harder” than it should be. But I am in complete agreement with the recommendation that economics needs to be more tolerant of different methods, whether the latest ultra-hard physics or the softer explorations of anthropology or even a business-school case study. After all, the economy encompasses a lot of different things; why should it yield only to a particular set of analytical tools?

Economics has certainly profited from the insights of those outside the field, such as psychologist Daniel Kahneman, and the late Elinor Ostrom, a political scientist whose ambition to study economics was thwarted because, as a girl in the 1940s, she’d been steered away from mathematics. Both have Nobel memorial prizes in economics.

The insights can flow the other way, too. Charles Darwin’s theory of evolution was influenced by the economist Robert Malthus’s Essay on the Principle of Population. When working on the kinetic theory of gases, physicist James Clerk Maxwell drew inspiration from social scientists and their habit of thinking in statistical approximations about large populations.

It is easier to make such a recommendation than to embrace it. The further the leading economics journals stray from their core expertise, the more difficult they will find it to distinguish good work from bad. But at the margin, such moves offer a lot of promise.

Written for and first published in the Financial Times on 28 June 2019.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.

How the Brexit debate was flushed down the drain

On a scale of one to seven, how well do you understand how a flush lavatory works?

This was a question asked by two Yale psychologists, Leonid Rozenblit and Frank Keil, almost two decades ago. Before I explain why, here’s a follow-up exercise: write down your lavatory explanation in as much detail as you can. You may wish to draw a diagram, or explain it to a friend. Or not.

You may then reflect that you knew a little less than you realised. That was the experience of many of the study’s subjects — and not just for lavatories (why does all the water disappear down the U-bend?) but also for zips, quartz watches, helicopters, speedometers, cylinder locks, piano keys and sewing machines. People felt they understood the mechanisms that surrounded them, but their confidence was severely dented by the simple act of giving them pencil and paper and saying: “Show me.”

The same exercise can be performed with politics. In 2013, Steven Sloman and Philip Fernbach, authors of The Knowledge Illusion, were members of a research team that did just that, inviting people resident in the US to rate their understanding of American policy proposals such as introducing unilateral sanctions on Iran, a cap-and-trade system for carbon emissions and a national flat tax. They also asked people to rate their approval of each policy, which would have been unnecessary for lavatories and zips. (Lavatories are useful, zips self-evidently malevolent.)

Professors Sloman and Fernbach and their colleagues found that — just as with locks and speedometers — people tended to overrate their knowledge at first, and then discover some humility when asked to be more specific.

Perhaps British voters could use a dose of the same medicine when it comes to our understanding of Brexit. Leave or Remain, many of us came late to the realisation that there was a difference between the single market and the customs union. I am still not sure most people can explain what that difference is. Many people have strong views about Prime Minister Theresa May’s withdrawal agreement; rather fewer could give a convincing account of how it differs from the political declaration that accompanies it.

When Mrs May began her premiership with the statement that “Brexit means Brexit”, it dawned on most of us that the details of the whole project might need a little more work. But she wasn’t the only one who was vague.

I’d love to see the contenders for the Conservative party leadership quizzed a little less about their cocaine habits and instead forced to sit down and write a detailed explanation of what a no-deal Brexit actually is. While we wait, perhaps the same exercise could be given to the 160,000 Conservative party members who are about to select the country’s next prime minister.

How long, for example, will HM Revenue & Customs wave through imports without inspections? Will the French reciprocate? What are the implications of “trading under World Trade Organization rules” for the UK’s banking and insurance industries? How large are those industries?

How many other developed countries are content to rely solely on WTO arrangements in their trade with the EU? Is the WTO capable of enforcing the rules anyway, given the current crisis in its appellate body? How likely is the EU to grant permission to British farmers to sell meat, milk or cheese? Would any of these decisions be different if the UK refused to pay the “divorce bill” it had negotiated?

I don’t think it is especially shameful that we ordinary voters are incurious about the ins and outs of Brexit, any more than we should be obliged to understand the workings of a quartz watch. An ability to read the time is generally sufficient. But I am stunned by just how little we seem to demand of our political leaders.

We want tailors to understand sewing machines, locksmiths to understand locks and plumbers to know that a lavatory is basically a siphon. But our standards for politicians seem far lower. The next prime minister is likely to be a person who believes that if we demanded it with enough gusto, sewage would remove itself from our homes in some scatological remix of Mary Poppins — and that anyone who tells you otherwise is clearly a shill for Big Porcelain.

We should expect more of anyone who wants to lead the country. And since our politicians have grown so fond of punting the hard questions back to us, perhaps we should also demand more of ourselves.

Profs Sloman and Fernbach found that asking people to explain the workings of the policies they so fervently supported or opposed had a humbling effect. When people realised that they knew less than they had once believed, they quite reasonably wound their necks in as a result. It seems strange to die in a ditch for something we can’t clearly explain, even to ourselves.

Next time you find yourself in some heated political debate, perhaps you should suggest that both sides pause to explain the policy in question. You may find you understand less — and agree more — than you realised.

Written for and first published in the Financial Times on 21 June 2019.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.

How the US is weaponising the world economy

Back in 2002, serious people were worried about the possibility of a nuclear exchange between India and Pakistan. Millions might have died — and the prospect seemed real enough that both the US and the UK advised their citizens to flee the region. How, then, was the crisis defused?

Thomas Friedman, author of The World Is Flat, is fond of telling the story that US businesses (in particular Dell) told their Indian suppliers (in particular Wipro) to calm things down or get cut out of the loop. And things did indeed calm down, so perhaps it was the concerns over Dell’s supply chain that prevented catastrophe. Perhaps.

Mr Friedman duly coined the phrase “the Dell Theory Of Conflict Prevention”: no two countries will go to war if they are part of the same global supply chain. He was never entirely serious about that, but the question now arises: did he have it backwards? Rather than a line of defence against hostile action, might global supply chains be a line of attack?

One example is the recent executive order banning US companies from working with Huawei, effectively denying the Chinese telecoms company the use of Qualcomm’s chips and Google’s Android operating system. Another was Mr Trump’s crude — and fleeting — threat to slap tariffs on Mexico if it didn’t satisfy him on immigration policy.

It is tempting to view such actions as uniquely Trumpian. Would any other president threaten sanctions against one of its largest trading partners, via Twitter, with 10 days’ notice, at the very moment they were presenting the new Trump-championed trade agreement to the Mexican senate?

Yet while a different president might act with more subtlety, the US seems unlikely to abandon the aggressive tweaking of the nerves and sinews under the skin of the world economy. Henry Farrell and Abraham Newman — political scientists at George Washington and Georgetown Universities, respectively — have popularised the term “weaponised interdependence”, the title of a forthcoming article in the journal International Security.

Messrs Farrell and Newman point out that supply chains and digital networks can be used both as a “panopticon” to see everything that happens and as a “chokepoint”, denying access to some vital service. Both approaches require a certain bureaucratic apparatus — something that would be hard to disassemble once in operation. There is more going on here than the whim of “Tariff Man”.

Consider Swift, the international financial messaging system. Although Swift does not directly handle transfers of money between banks, it provides the secure service that makes those transfers possible. Swift is a private company based in Brussels, yet last summer it found itself on the receiving end of US demands to cut off Iranian banks. The EU, in turn, demanded that it did not comply. Forced to pick a side in this tug of war, late last year it picked the US.

That is an indication of just how much power the US can wield if it is determined to do so. And the temptation is strong: it seems far safer to attack Iranian interests through stern letters to a messaging service in Brussels than with a carrier strike group.

That very temptation, of course, risks over-reach. The US is not the first global superpower to ponder the use of financial and communication networks as a weapon of war. In the early 20th century, modern economies were increasingly underpinned by complex financing. Britain viewed the central role of the City of London in the world’s banking, telegraph and marine insurance system as potentially decisive when coupled with the power of the Royal Navy. Should war break out with Germany, these networks could be used to sustain the UK economy while crushing that of Germany.

The idea seemed plausible, but needless to say, these plans for economic shock and awe failed both to head off the first world war and to limit its duration and brutality.

Two questions arise: would the US be wise to use its economic leverage more sparingly? And should other nations be building alternative networks beyond the hegemon’s gaze and grip?

It is too easy to say that the US should restrain itself in its own enlightened self-interest. The logic of network effects is self-reinforcing. Having established a central position in finance through Wall Street and the mighty dollar, and in digital networks thanks to Silicon Valley and the Pentagon’s role in funding the early internet, the US advantage may endure many abuses.

Still, the more the US seeks to coerce others through its privileged position in banking and the internet, the greater the incentive to develop alternatives that cut the US out of the loop — for example, a Chinese-built operating system for smartphones, or Instex, the special purpose vehicle launched by France, Germany and the UK to allow companies to do business with Iran beyond the reach of US punishment.

Wolfgang Munchau described Instex as “a dysfunctional insurance vehicle for small carpet traders”; it turns out that these alternatives are not cheap or easy to develop. In ordinary circumstances, few would even bother to try. But if the US presses too painfully on the global economy’s nervous system, its rivals and even its allies will look for relief.

Written for and first published in the Financial Times on 14 June 2019.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.

What to do when blessings come well-disguised

Keith Jarrett’s 1975 concert in Cologne should have been a musical catastrophe. Owing to a string of mix-ups and bad luck, he was faced with the choice of attempting his widely admired improvisations on a beaten-up old piano with sticky keys and a harsh upper register — or walking out altogether. He was all for walking out, but felt sorry for the concert promoter and agreed to play the unplayable piano against his better judgment. The result was not a catastrophe but a masterpiece, and a bestselling one at that: The Köln Concert album.

I’m fond of that story, of the way that an obstacle can unleash a creative response — partly by concentrating the mind and partly by forcing the artist to explore fresh approaches. It’s not the only such tale. The great jazz guitarist Django Reinhardt developed his distinctive style after suffering severe burns that left him with only two fully functioning fingers on his fretting hand.

A quotidian parallel is the Tube strike of 2014, which partially shut down the London Underground, prodding several thousand commuters into finding better routes into work.

The three stories share another common element: nobody at the time would have predicted that anything good would come of the problem. Keith Jarrett could have sought out a bad piano; young Django could have tied two fingers together; commuters could have tried a new route any day. But they didn’t, until there was no choice. The blessings came well disguised.

David Epstein’s new book, Range, has brought other instances to my attention. Range is mostly about the benefits of being a generalist rather than a specialist, but a recurring theme is of teaching or training techniques that seem to fall flat but pay off in the long run.

For instance, in the US Air Force Academy, students are initially assigned at random to academic instructors, then later randomly reassigned to follow-on courses — a nice natural experiment. Scott Carrell and James West looked at the data and found snatches of unplayable piano in it: students assigned the “best” instructors for introductory courses, as measured by short-term pupil performance, earned the highest student evaluations but went on to produce the worst results in the long run.

In a nutshell, teachers who “teach to the test”, or otherwise provide simple problem-solving procedures, immediately improve grades and their students thank them for it. But in the long-term, the students suffer from not having been forced to think more deeply.

This isn’t just about pedagogical techniques. One study that stuck with me when I was researching my own book, Messy, was by Katherine Phillips and other psychologists. The researchers set small groups working on a problem before introducing a new team member to help, sometimes a stranger and sometimes a familiar face. The groups forced to work with the stranger were much more likely to solve the problem, but also enjoyed the experience less and sharply underrated their success. Groups of friends did much worse, but had fun and were under the illusion that they had done a good job.

Again the striking thing is not just that the obstacle turned out to be helpful, but that nobody thought so at the time. Only with hindsight did people realise.

Three more examples. Sports scientists now suggest that top endurance athletes should keep most of their training at low intensity, resisting the temptation to train too hard. Traffic planners are familiar with Braess’ Paradox: closing a road can and often does improve traffic flow, even if the number of journeys stays the same. The closure prevents drivers from choosing routes that make sense for them but cause congestion for others.

The inverse is even more common: a pleasing intervention that does no good. For instance, preliminary research suggests that using virtual reality to promote empathy — for example, for refugees, homeless people or those with disabilities — can promote a surge of emotional support in the short run without delivering any long-run insight.

We shouldn’t leap to the conclusion that every blessing comes well disguised, and that every problem is an opportunity. Sometimes blessings are perfectly apparent, and we should remember to count them. Sometimes a problem is simply a problem. But the prevalence of these counterintuitive results is a reminder that we know less about the world — and about our reaction to it — than we like to think.

I draw three lessons. The first is the need to gather solid evidence — for example, by running randomised trials of teaching techniques — and to ensure that the evidence looks at a good range of outcomes over a long enough time to be meaningful. Common sense can lead us astray, especially in situations where short-term pain leads to long-term gain.

The second lesson is to experiment with more variety in our own lives — whether choosing a holiday or a commuting route. It shouldn’t take industrial action to prompt us to try an alternative route to work.

The final lesson is the simplest. When life confronts us with an unplayable piano, perhaps we should sit down and try to play it.

Written for and first published in the Financial Times on 7 June 2019.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.

The dying art of compromise

I don’t often find myself agreeing with Esther McVey, but I wondered this week whether the candidate for leader of the UK Conservative party might accidentally have spoken the truth: “People saying we need a Brexit policy to bring people together are misreading the situation. That is clearly not possible.”

The British do indeed seem in no mood to compromise. The results of elections to the European Parliament produced a thunderous endorsement of parties that proudly reject an attempt to find common ground on Brexit. The Conservatives and Labour, each caught in an awkward straddle, were slaughtered. Labour offered the slogan “let’s bring our country together”. Ha! Voters preferred the Liberal Democrats (“Bollocks to Brexit”) and the Brexit party (“they’re absolutely terrified of us”).

Sometimes an extreme position is the correct one. When King Solomon proposed cutting the baby in half, it wasn’t because he was looking for the middle ground.

Yet a capacity to find compromises is a good thing to have. Positions may differ, but whether we live in the same home or on the opposite side of the planet, we benefit when we can find a way to get along.

If this new distaste for compromise is a problem, it is not the UK’s alone. Positions seem to be hardening everywhere, the sclerotic arteries that may lead to a heart attack for western democracies. Perhaps this is driven by personalities. For a man whose name adorns a book titled The Art of The Deal, Donald Trump is curiously uninterested in negotiating lasting agreements with anyone. Or maybe it is a function of an information ecosystem in which outrage sells.

Perhaps the problems themselves are more intractable. Some issues do not lend themselves to compromise. Brexit is one. Splitting the difference between Remainers and hard Brexiters is less like cutting a cake and more like splattering its ingredients everywhere. Egg on my face, flour on yours, and nobody even partially satisfied.

Abortion is another. There is a principled case to be made for a woman’s absolute right to control her body. There is also a principled case to be made for the absolute right to life of a foetus. But like the unstoppable cannonball and the immovable post, both rights cannot be absolute simultaneously.

In contrast, other complex and emotive problems may still allow for compromise. On climate change, we can shrug and do nothing, or we can turn our economic system upside down, but there is plenty of middle ground between those options. In a trade negotiation, a mutually advantageous outcome is almost always there to be discovered.

Roger Fisher and William Ury’s classic negotiation handbook Getting to Yes advises: focus on the problem rather than the personalities; explore underlying interests rather than explicit positions; and consider options that may open up scope for mutual benefit. We may find a much better way to split the cake if we discover that you scrape the icing into the bin, while I would happily eat it with a spoon. It is sometimes astonishing how far a principled negotiation can go towards giving both sides what they want.

It is clear that we British have failed to follow this advice. Our debate is driven by a bitter focus on personalities, from Theresa May to Nigel Farage to Jeremy Corbyn to the generic “Remoaner elite”. Each side knows what the other wants but has shown very little interest in why they want it. Without sincerely exploring the underlying aims and values of warring tribes there is no chance of finding an outcome everyone can accept.

The US debate also seems the antithesis of Fisher and Ury’s advice. Too many politically active people seek the humiliation of the other tribe. Dismissing compromise as craven appeasement seems to be a winning tactic, particularly in the primary elections that set the tone of US politics.

Compromise, however, is often possible even in unpromising situations. On abortion, for example, it emerges with a focus not on absolute rights but on practicalities. Many people can get behind policies to minimise unwanted pregnancies, and to make abortions safe and regulated rather than dangerous and illicit. It is a middle ground that many countries manage to find.

One can see politics as a competitive sport or a search for solutions. There’s truth in both views. However, a democratic election is far closer to a competition than to a principled negotiation. Do we not wish to see the opposite team soundly thrashed? Do we not boo their villainous antics and laugh at their mishaps? Who wants to play out a nil-nil draw?

I would not want to venerate compromise as the supreme good in politics. Sometimes it really is true that you and I, dear reader, are absolutely right and they are absolutely wrong. (It may even be true that we are absolutely wrong and they are absolutely right.) Either way, the merits of the case must be weighed against the merits of trying to respect everyone. It feels good to win, but this isn’t a fairytale: the losers won’t stamp their feet and vanish through the floor. They — or we — aren’t going anywhere.

Written for and first published in the Financial Times on 31 May 2019.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.

Why brilliant people lose their touch

It hasn’t been a great couple of years for Neil Woodford — and it has been just as miserable for the people who have entrusted money to his investment funds. Mr Woodford was probably the most celebrated stockpicker in the UK, but recently his funds have been languishing. Piling on the woes, Morningstar, a rating agency, downgraded his flagship fund this week. What has happened to the darling of the investment community?

Mr Woodford isn’t the only star to fade. Fund manager Anthony Bolton is an obvious parallel. He enjoyed almost three decades of superb performance, retired, then returned to blemish his record with a few miserable years investing in China.

The story of triumph followed by disappointment is not limited to investment. Think of Arsène Wenger, for a few years the most brilliant manager in football, and then an eternal runner-up. Or all the bands who have struggled with “difficult second-album syndrome”.

There is even a legend that athletes who appear on the cover of Sports Illustrated are doomed to suffer the “SI jinx”. The rise to the top is followed by the fall from grace.

There are three broad explanations for these tragic career arcs. Our instinct is to blame the individual. We assume that Mr Woodford lost his touch and that Mr Wenger stopped learning. That is possible. Successful people can become overconfident, or isolated from feedback, or lazy.

But an alternative possibility is that the world changed. Mr Wenger’s emphasis on diet, data and the global transfer market was once unusual, but when his rivals noticed and began to follow suit, his edge disappeared. In the investment world — and indeed, the business world more broadly — good ideas don’t work forever because the competition catches on.

The third explanation is the least satisfying: that luck was at play. This seems implausible at first glance. Could luck alone have brought Mr Wenger three Premier League titles? Or that Mr Bolton was simply lucky for 28 years? Do we really live in such an impossibly random universe?

Perhaps we do. Michael Blastland’s recent book, The Hidden Half, argues that much of the variation we see in the world around us is essentially mysterious. Mr Blastland’s opening example is the marmorkrebs, a kind of crayfish that reproduces parthenogenetically — that is, marmorkrebs lay eggs without mating and those eggs develop into clones of their mothers.

Place two clones into two identical fish tanks and feed them identical food. These genetically identical creatures raised in apparently identical environments produce genetically identical offspring who nevertheless vary dramatically in their size, form, lifespan, fecundity, and behaviour. Sometimes things turn out very differently for no reason that we can discern. We might as well call that reason “luck” as anything else.

This is not to say that skill doesn’t matter — merely that in a competition in which all the leaders are highly skilled, randomness may explain the difference between triumph and failure. Good luck plus skill beats bad luck plus skill any time.

It is easy to underestimate how much chance is at play all around us. The psychologist Daniel Kahneman has recently been studying what he calls “noise”: the variability of judgments for no obvious reason.

A wine expert blind-tasting two glasses from the same bottle will often rate them differently. Pathologists disagree with each other in their judgments of the same biopsy. More disconcertingly, they also disagree with their own prior judgments of the case.

We rarely appreciate just how much inconsistency there is in the judgments we and others make, argues Prof Kahneman. It can hardly be a surprise, then, if past performance is no guarantee of future success.

We should remember, too, that people often achieve outsized success by taking risks or being contrarian. When John Kay examined the forecasting record of economists in the 1990s, he noted that Patrick Minford, an idiosyncratic forecaster, would often produce the best forecast one year and the worst forecast the next. If the consensus is wrong, being an outlier gives you a high chance both of dramatic success and spectacular failure.

We perceive all this randomness through a particular filter, too. Few people make the cover of Sports Illustrated after a run of mediocre luck. They appear after things have been going well, and if the good luck fails to hold then it seems like the SI jinx. More likely it is “regression to the mean”, or in simple terms, a return to business as usual.

We begin paying attention only when someone is producing a remarkable performance. Genius followed by mediocrity is a story arc we all notice. Mediocrity followed by genius just looks like genius — assuming the mediocre performer gets a second chance. Not all do.

So I wish Mr Woodford well. Perhaps he has lost his touch, perhaps the world has changed, or perhaps he has simply been unlucky. It would be nice to know which, but in such matters the world does not always satisfy our curiosity.

Written for and first published in the Financial Times on 24 May 2019.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.

The Doris Day effect – when obstacles help us

She had hoped to become a ballet dancer. After her leg was shattered in an accident at the age of 15, she took singing lessons instead. It was a striking detail in the obituaries. If not for that painful setback, the star that was Doris Day would never have risen.

Was the car accident that redirected her career an extraordinary twist in the story of an extraordinary life? Or was it typical of some broader truth about life, that frustrations can actually help us? Perhaps it is true that what does not kill us makes us stronger. It may, in contrast, be that what does not kill us nevertheless slows us down.

The conventional wisdom is that initial advantages tend to snowball into an avalanche of privilege. Sometimes this reflects genuine achievements: a bit of luck with an early teacher sharpens a student’s skills, lifting her into a higher set, which in turn gets her into a better university, then a job with more stimulating peers, and so on.

An egregious example, made famous by Malcolm Gladwell’s book Outliers, is the tendency of elite athletes to be born early in their school year. Being a few months older at the age of five means you are stronger and faster, are more likely to be picked for school teams, get more practice and are still reaping the benefits as an adult athlete. The effect is particularly well-studied among boys playing ice hockey in Canada, and football in a variety of countries.

At other times, well-deserved acclaim is followed by unearned praise. In academia this tendency was named by the sociologist Robert K Merton as “the Matthew Effect” in reference to a biblical verse: “For to every one who has will more be given, and he will have abundance; but from him who has not, even what he has will be taken away.”

If three researchers collaborate on a problem, and one of them already has a Nobel Prize, the laureate tends to earn disproportionate recognition for the joint work. When a teacher and a student work together, the senior researcher is cited because that name is already recognised. The junior is easily forgotten.

In the wider workplace, we have evidence that the luck of graduating in a benign economic climate can lead to a lasting advantage. One researcher, Paul Oyer, found that young PhD and MBA students who started off in favourable job markets were employed in better places with smarter colleagues, and were still doing better a decade later than those who started out in tougher times.

Hannes Schwandt and Till Marco Von Wachter studied the other end of the US labour market to find the story is even worse there: entering the job market during a recession damages anyone’s prospects, but the harm is deeper and lasts longer for less-educated and otherwise disadvantaged groups.

All this suggests that setbacks are setbacks: they drag us down, perhaps disproportionately. Doris Day was an exception, not the rule.

Yet a striking new study suggests that the Doris Day effect is quite real in one particular group of people: young scientists applying for research grants. Yang Wang, Benjamin Jones and Dashun Wang looked at scientists applying for funding from the US National Institutes for Health, with grants averaging $1.3m. In particular, they focused on borderline decisions, comparing those who scraped through to get a grant with those who just missed out. The near-winners and the near-losers were otherwise indistinguishable before the decision point, but afterwards it was the losers who prospered, publishing substantially more highly cited research papers.

We should remember that anyone in a position to nearly secure a million-dollar research grant has presumably enjoyed a few successes along the way at school and university. Failure at this hurdle may be described as an “early career setback”, but it is not comparable to the setback suffered by an undernourished two-year-old with no books in her bedroom.

Still: this is a counterintuitive finding. Yet I was not entirely surprised to encounter it. It may be that many people respond to a setback by bouncing back with renewed determination. It may also be that the failure provokes a rethink and a fresh course of action. Doris Day, after all, did not respond to a shattered leg by trying even harder to become a dancer. She changed her goals and prospered as a result.

We don’t have to be promising young scientists — or aspiring starlets — to benefit from having obstacles placed in our way. Something as mundane as a strike disrupting regular commuting has been shown to push people towards new habits. Three economists who studied data from London’s public transport network found that after a 48-hour strike in London in February 2014, thousands of commuters changed routes and never switched back. They discovered that they’d been doing the commute wrong their entire working lives.

Often failure is simply failure, and a setback is exactly what it seems. But sometimes the obstacle that has been placed in our path might provoke us to look around, and perhaps to discover that a better route was there all along.

Written for and first published in the Financial Times on 17 May 2019.

My book Messy explores some of these ideas in more depth – feel free to order online or through your local bookshop.

Why we should favour second guesses over first instincts…

Tension is rising in the Harford household as exams approach and we try to persuade Miss Harford Sr to relax, and Miss Harford Jr to be slightly less relaxed. I’m sure many readers have vivid memories of the exam room, recent or otherwise. But here’s a question about exam technique that suggests a much wider lesson. In a multiple-choice test, you sometimes write down an answer and then have second thoughts. Is it wise to stay with your first instincts, or better to switch?

Most people would advise that the initial answer is usually better than the doubt-plagued second guess. Three-quarters of students think so, according to various surveys over the years. College instructors think so too, by a majority of 55 to 16 per cent. The 2000 edition of Barron’s How to Prepare for the GRE Test is very clear that students should be wary of switching. “Experience indicates that many students who change answers change to the wrong answer.”

This confidence would be reassuring, were it not utterly erroneous. Researchers have been studying this question since the 1920s. They have overwhelmingly concluded both that individual answer changes are more likely to be from wrong to right, and that students who change their answers tend to improve their scores. This gap between perception and reality is stark enough to have earned a name: the “first-instinct fallacy”. No doubt our first instincts are often right, but when we start to have second thoughts, the second thoughts are usually occurring for a reason. It is better to switch. So why don’t we?

Justin Kruger, a psychologist at New York University, has been studying this question. (Prof Kruger is more famous as co-discoverer of the Dunning-Kruger effect: people who are incompetent are too incompetent to realise how incompetent they are.) With his colleagues Derrick Wirtz and Dale Miller he replicated the longstanding findings that college students believe you should trust your first answer in a multiple choice question, and yet that switching to a second answer tends to improve your grades. Then the trio started to explore why.

In one study they showed subjects video based on the TV show Who Wants To Be A Millionaire? and asked them to imagine that they were watching a teammate play, accumulating cash for the team whenever he or she gave the correct answer. Some subjects were shown teammates who always switched on 50/50 questions, while others were shown teammates who always stuck with their first instinct. In the study, both strategies produced identical results, yet subjects watching a switching teammate were more frustrated and critical and had a good memory for the errors.

Another study by Prof Kruger and his colleagues showed that we also have a warped recollection of our own errors in multiple choice tests. We have a rosy memory of sticking to our first instincts, forgetting the failures and exaggerating the successes. We vividly recall switching to the wrong answer and overestimate how often we did so. In short, we remember sticking as having been the best tactic, when in fact switching was better. No wonder the Barron’s guide remarks that “experience” tells us switching is a bad idea. Our own experiences do indeed tell us that, but only because we misremember the lessons of previous switches.

If you — or a loved one — are about to enter exam season, perhaps this evidence-based strategy will be of use. But it’s hard not to see a broader lesson. How often in life do we make a choice and then stick to it despite mounting doubts?

In politics, such questions are aggravated by questions of partisanship and pride. Nobody wants to admit that they were wrong in the face of jeers from those on the opposite side of the political fence. The U-turn is one of the greatest sins in politics, if only because it is so easy to criticise. Either you were wrong before or you are wrong now.

But even in everyday life, we find ourselves clinging to bad choices. Steven Levitt, the co-author of Freakonomics, once conducted a study in which people hesitating over big choices — to leave a spouse, to adopt a child, to quit a job, to start a business — agreed to be guided by a coin toss. Those who had been nudged to act ended up being happier several months on than those who had been nudged to stick with the status quo.

We are prone to cling tightly to the devil we know. The likely explanation is that we are seeking to minimise regret. Forget the old cliché about regretting the things you didn’t do more than the things you did: we regret misfiring action more than misfiring inaction. We starkly remember the times we changed things for the worse, and we more easily forget the times when we failed to change things for the better. It’s not that impulsive action is always the best option, any more than your first answer on a test is always wrong. Instead, the lesson is that if you are hesitating over whether to leave things as they are, you probably needed to make a change some time ago.

Written for and first published in the Financial Times on 10 May 2019.

My book “Fifty Things That Made the Modern Economy” (UK) / “Fifty Inventions That Shaped The Modern Economy” (US) is out now in paperback – feel free to order online or through your local bookshop.