Why real life needs real trials

Finding out what works is a serious business and this is a brilliantly simple way to do less harm and save money

I sometimes feel that if only politicians and civil servants had a “Ladybird Book of Randomised Policy Trials”, they’d understand why such trials were a brilliantly simple way for Whitehall to do more good, less harm and save money.

Well, I have hugely encouraging news to report: the “Ladybird Book of Randomised Policy Trials” now exists. Better than that, it’s not just some academic flight of fancy. It is the result of a collaboration between two academic experts, Professor David Torgerson and Dr Ben Goldacre, and two civil servants, Laura Haynes and Owain Service, of the Cabinet Office’s Behavioural Insights Team. In other words, something might actually happen.

My enthusiasm is scarcely dented by the fact that the document is actually called “Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials”; no Ladybird, alas, although Goldacre tells me he wishes that had been the title.

Randomised trials are important in policy work for exactly the same reason that they are important in medicine: because our theories about what works are often badly wrong, and in a complex world it is often only a properly controlled trial that will set us straight.

A good example in medicine is the use of steroid injections for people with head injuries, to reduce swelling and brain damage. The theory was sound – so sound that many doctors argued that it was unethical to conduct a randomised trial. Yet when the trial was eventually run and published, in 2005, it turned out that the well-meaning steroid injections had been inadvertently killing people.

There are similar examples among the few randomised trials of policy. “Scared straight” programmes, designed to warn juvenile delinquents of the consequences of a life of crime, turn out to cause crime, alas.

You might think it is far harder to run a trial on a social programme than in medicine, but Torgerson, director of the York Trials Unit, tells me the opposite is true. Clinical trials are difficult – patients often wander off or drop out. By contrast, many of the trials one might do in a school or prison involve people who were going to be there anyway.

The difference is in our priorities. Torgerson points out that he recently spent £500,000 testing verruca treatments; for the same money he could find out whether phonics is an effective method for teaching children to read.

So why is it so hard to get funding, and approval, for randomised policy trials? One common objection is that we don’t need trials because we already know what works. This is nonsense, of course: very often different experts or political parties “know” things that are mutually contradictory.

A second objection is that trials are expensive to run. Perhaps; although not as expensive as blundering around without a clue, and trials are often cheaper to run than one might think.

Policies are often rolled out gradually; if this gradual roll-out is randomised, drawing conclusions from the new policy becomes vastly simpler and more robust.

Perhaps the real problem is that too many politicians and civil servants have little knowledge of, or respect for, basic scientific methods. According to Mark Henderson, author of The Geek Manifesto, a mandarin at the Department for Work and Pensions once told Tony Blair’s chief scientific adviser that the DWP could function perfectly well without any contribution from science – a demonstration of grotesque ignorance and arrogance. Fortunately, as the new guide explains, the DWP is already benefiting from evidence gleaned from randomised trials.

What matters is what works, we have been told by generations of politicians. But finding out what works is a serious business. Perhaps, at last, it is also a business that Whitehall is taking seriously.

Also published at ft.com.

30th of June, 2012 • Undercover Economist • Comments off