Failure: It’s everywhere

I wrote this essay for the Freakonomics blog.

In 1982, the management consultants Tom Peters and Robert Waterman published In Search of Excellence, a colossally popular business title. The book aimed to learn lessons from the world’s best companies, and Peters and Waterman produced a list of 43. But just a couple of years after In Search of Excellence had been published, BusinessWeek ran a cover story with the simple title: “Oops! Who’s Excellent Now?” Almost a third of the companies singled out for praise by Peters and Waterman were in financial trouble.

My aim isn’t to mock Peters and Waterman, but to point out that the rise and fall of business models is an unavoidable part of economic growth. In a complex world, things fail – a lot. According to the economist Paul Ormerod, 10 percent of U.S. firms go bankrupt every year. Ormerod – an iconoclastic figure who enjoys beating fellow economists at their own game, mathematics – has studied the statistical patterns that emerge from these bankruptcies. He thinks they suggest that failure and success in business are far more random than our culture of CEO-worship would have us believe.

The curious thing is that these individual failures need not be a problem for the economy as a whole. Far from it. There are good reasons to believe that more successful economies play host to more failures. To get an instinctive grasp of this, just compare the astonishing failure rate of Silicon Valley firms with the situation of the Big 3 in Detroit, who seem to be in a perpetual state of slow-motion failure without ever quite leaving the economic stage.

A more rigorous attempt to look at this question, a study by Kathy Fogel, Randall Morck,and Bernard Yeung, found statistical evidence that economies with more churn in the corporate sector also had faster economic growth. The relationship even seems causal: churn today is correlated with fast economic growth tomorrow. The real benefit of this creative destruction, say Fogel and her colleagues, is not the appearance of “rising stars” but the disappearance of old, inefficient companies. Failure is not only common and unpredictable, it’s healthy.

But if this is true, it makes nonsense out of much of the way we approach complex problems in the world – anything from fighting wars to fighting poverty.

For one thing, where’s the churn in education policy or healthcare policy or policing? These are difficult problems. Why would we expect them to be solved the first time? They are surely no simpler than the business problems which seem so prone to experiment and error.

When the pizza chain Sbarro’s filed for bankruptcy in April, one journalist called it “America’s least essential restaurant.” The likely result is that one way or another, the chefs and locations and ingredients that used to be turned into bland pizzas will now be used to make something that people actually want to pay for. Good.

But in politics, where are the bad ideas that have been tested, found wanting, and replaced with something better? It’s rare – but not unheard of – for politicians to seriously test out their policies, perhaps because they realize that we voters pay more attention to soundbites. And so there’s rarely a really good evidence base to shut down failing policies and replace them with something else.

I’m not sure I’d blame the politicians. Not many people want to vote for a candidate who says, “I really am not sure how to improve schools in our area, so I plan to pilot half a dozen ideas, and we’ll keep the one or two that we can prove to have worked.” But the other guy – the one who claims he does know – isn’t telling us the truth about uncertainty and failure. He’s just telling us what we want to hear.

Ideally, I’d like to see many more complex problems approached with a willingness to experiment. The process has three components: first, try lots of different things; second, make sure the experiments are at a small scale so that when things go wrong, it’s not a catastrophe; and third, make sure there’s a reliable way to tell the difference between success and failure.

Governments often fall down on all three: they have a particular ideology and so push a single-minded policy; they bet big; and they don’t bother to evaluate the results too carefully, perhaps through overconfidence.

But markets can fail badly too, and for much the same reason. Just think about the subprime crisis. It failed the same three tests. First, many big banks and insurance companies were taking similar bets at similar times, so that when subprime loans started to go bad, much of Wall Street started struggling simultaneously. Second, the bets were gigantic. Fancy derivatives such as credit default swaps and complex mortgage-backed securities were new, rapidly growing, and largely untested. And third, many investment bankers were being paid large bonuses on the assumption that their performance could be measured properly – and it couldn’t, because profitable-seeming bets concealed large risks.

I’ve been studying how these processes of experimenting and adapting have worked, or failed to work, in dealing with some of the important problems we face today, from promoting innovation to preventing future financial meltdowns. And I’ve also been exploring our psychological responses to failure – why we fear even small setbacks, and how we can over-react when things are going badly. I’ve concluded that adapting in the face of failure is absolutely vital – and a more painful challenge than I at first expected.

Nowhere is that more true than on the battlefield, and in my next post I’ll be exploring how the military learns from failure – from the Vietnam War to the hunt for Bin Laden.

 

12th of May, 2011 • Other Writing • Comments off