Why pilot schemes help ideas take flight
There’s huge value in experiments that help us decide whether to go big or go home
Here’s a little puzzle. You’re offered the chance to participate in two high-risk business ventures. Each costs £11,000. Each will be worth £1m if all goes well. Each has just a 1 per cent chance of success. The mystery is that the ventures have very different expected pay-offs.
One of these opportunities is a poor investment: it costs £11,000 to get an expected payout of £10,000, which is 1 per cent of a million. Unless you take enormous pleasure in gambling, the venture makes no sense.
Strangely, the other opportunity, while still risky, is an excellent bet. With the same cost and the same chance of success, how could that be?
Here’s the subtle difference. This attractive alternative project has two stages. The first is a pilot, costing £1,000. The pilot has a 90 per cent chance of failing, which would end the whole project. If the pilot succeeds, scaling up will cost a further £10,000, and there will be a 10 per cent chance of a million-pound payday.
This two-stage structure changes everything. While the total cost is still £11,000 and the chance of success is still 1 per cent, the option to get out after a failed pilot is invaluable. Nine times out of 10, the pilot will save you from wasting £10,000 – which means that while the simple project offers an expected loss of £1,000, the two-stage project has an expected profit of £8,000.
In a real project, nobody could ever be sure about the probability of success or its rewards. But the idea behind this example is very real: there’s huge value in experiments that help us decide whether to go big or go home.
We can see this effect in data from the venture capital industry. One study looked at companies backed by US venture capitalists (VCs) between 1986 and 1997, comparing them with a sample of companies chosen randomly to be the same age, size and from the same industry. (These results were published in this summer’s Journal of Economic Perspectives in an article titled “Entrepreneurship as Experimentation”.)
By 2007, only a quarter of the VC-backed firms had survived, while one-third of the comparison group was still in business. However, the surviving VC-backed firms were big successes, employing more than five times as many people as the surviving comparison firms. We can’t tell from this data whether the VCs are creating winners or merely spotting them in advance but we can see that big successes on an aggregate scale are entwined with a very high failure rate.
The option to conduct a cheap test run can be very valuable. It’s easy to lose sight of quite how valuable. Aza Raskin, who was lead designer for the Firefox browser, cites the late Paul MacCready as his inspiration on this point. MacCready was one of the great aeronautical engineers, and his most famous achievement was to build the Gossamer Condor and the Gossamer Albatross, human-powered planes that tore up the record books in the late 1970s.
One of MacCready’s key ideas was to develop a plane that could swiftly be rebuilt after a crash. Each test flight revealed fresh information, MacCready figured, but human-powered planes are so feather-light that each test flight also damages the plane. The most important thing a designer could do was to build a plane that could be rebuilt within days or even hours after a crash – rather than weeks or months. Once the problem of fast, cheap experimentation was solved, everything else followed.
Some professions have internalised this lesson. Architects use scale models to shed light on how a completed building might look and feel. A nicely made model can take days of work to complete but that is not much compared with the cost of the building itself.
Politicians don’t find it so easy. A new policy is hardly a new policy at all unless it can be unveiled in a blaze of glory, preferably as a well-timed surprise. That hardly suits the MacCready approach. Imagine the conference speech: “We’re announcing a new array of quick-and-dirty experiments with the welfare state. We’ll be iterating rapidly after each new blunder and heart-rending tabloid anecdote.”
A subtler problem is that projects need a certain scale before powerful decision makers will take them seriously.
“The transaction costs involved in setting up any aid project are so great that most donors don’t want to consider a project spending less than £20m,” says Owen Barder, director for Europe at the Center for Global Development, a think-tank. I suspect that the same insight applies far beyond the aid industry. Governments and large corporations can find it’s such a hassle to get anything up and running that the big stakeholders don’t want to be bothered with anything small.
That is a shame. The real leverage of a pilot scheme is that although it is cheap, it could have much larger consequences. The experiment itself may seem too small to bother with; the lesson it teaches is not.
Also published at ft.com.