It may seem mind-bendingly obvious but we need to test and evaluate ideas
How do we keep young people away from a life of gangs and violent crime? You can see one answer if you fire up YouTube and type “Scared Straight” into it. You’ll have the pleasure of seeing muscular prisoners bully terrified teenagers while police officers stand by and watch. It’s an American reality TV show called Beyond Scared Straight and it’s into its seventh season.
Before Beyond Scared Straight there was Scared Straight. It was a 1978 documentary about a crime prevention programme of the same name, in which teenagers spent a day inside a prison being frightened by the inmates. The film was presented by Peter Falk at the height of his fame as Columbo, and it won an Oscar for best documentary. Its producer-director, Arnold Shapiro, went on to make the US version of Big Brother and Beyond Scared Straight.
The Scared Straight approach is popular as TV, and seems popular as public policy. But while Scared Straight was a success as a documentary, Scared Straight is a failure as a policy. We know this because, on seven occasions, administrators have allowed the programme to be evaluated rigorously using a controlled trial. Some troubled teens experienced the joys of Scared Straight while others did not, allowing a fair test of the programme’s results.
These seven rigorous evaluations form the foundation of a review of Scared Straight (and similar interventions) by the Cochrane Collaboration. The review concludes that “programmes such as Scared Straight increase delinquency relative to doing nothing at all to similar youths. Given these results, we cannot recommend this programme as a crime prevention strategy.”
If there is a question mark over a programme’s effectiveness then we need to make sure we’re not wasting effort, or even causing harm without meaning to. We should routinely test ideas, and adapt them if they’re not working. If that might seem a mind-bendingly obvious idea, let’s compare it with how programmes are evaluated in reality.
Consider Your Life You Choose (YLYC), a programme involving magistrates, police and prison officers which began in Ealing, west London, and which aims to reach 11- to 12-year-olds and steer them away from a life of crime. It hasn’t yet been rigorously evaluated.
Oddly, media reports seem to wish that YLYC was like Scared Straight, even though Scared Straight does not work. A recent headline in the London Evening Standard described YLYC as “Schoolchildren in north London taught lessons on life behind bars: Children put in handcuffs and prison van in anti-gang drive.” The article was accompanied by cheesy photographs of 11-year-olds in … well, handcuffs and a prison van.
Despite the Evening Standard’s enthusiasm for such photos, YLYC bears a blessedly superficial resemblance to Scared Straight – it’s delivered in schools, not prisons. Pam Ullstein, the YLYC project leader, says the handcuffs and van may “provide some entertainment” but are not the point of the programme. Good.
So YLYC might indeed work, and it might not. It would be wonderful to find out. Time for an evaluation?
Alas, the government’s leading authority on the matter, Damian Green – who this week lost his post as the Minister for Police and Criminal Justice – seemed to think no evaluation was needed. “Official figures demonstrate that it really is having an impact,” he said of YLYC in a speech in March.
I asked the Ministry of Justice what Green had in mind when he said this. I was directed to a page on the YLYC website itself, in which a police sergeant observes that in Ealing, youth convictions have fallen sharply in recent years. (Oddly, the web page also features Ealing’s Conservative MP, Angie Bray, praising YLYC with exactly the same words as Green: “Official figures demonstrate that it really is having an impact.”)
. . .
The fall in crime is good news, and perhaps that’s what Green and Bray mean by “official figures”. Yet youth convictions have also been falling sharply in England as a whole, so perhaps it’s a coincidence. If we are to have serious evidence on YLYC – or any other programme – we need rigorous evaluations, not a nod towards “official figures” of passing relevance.
The sociology of this is fascinating: we have an unproven programme that politicians are happy to praise and that a newspaper admires for its faint resemblance to a proven failure.
None of this is a criticism of YLYC, which may indeed be effective. We’re fortunate that people want to set up programmes such as YLYC and volunteer to support them. But we don’t want them to waste their time, so we should provide help in figuring out whether what they do actually works.
The good news is that help is available. There’s Project Oracle, for example, a new London-focused outfit that aims to support programme providers in gathering useful evidence about what’s working, while also educating the people who commission such programmes that it’s important to ask for evidence. Professor Georgie Parry-Crooke, co-director of the project, tells me that an evaluation of the effectiveness of Project Oracle itself is on the cards. That’s all to the good: reality TV is no basis for figuring out what works and what doesn’t. The evidence revolution will not be televised.
Also published at ft.com.