Review of The Tyranny of Metrics by Jerry Muller
Jerry Z Muller’s latest book is 220 pages long, not including the front matter. The average chapter is 10.18 pages long and contains 17.76 endnotes. There are four cover endorsements and the book weighs 421 grammes. These numbers tell us nothing, of course. If you want to understand the strengths and weaknesses of The Tyranny of Metrics (UK) (US) you will need to read it — or trust the opinion of someone who has.
Professor Muller’s argument is that we keep forgetting this obvious point. Rather than rely on the informed judgment of people familiar with the situation, we gather meaningless numbers at great cost. We then use them to guide our actions, predictably causing unintended damage.
A famous example is the obsession, during the Vietnam war, with the “body count” metric embraced by US defence secretary Robert McNamara. The more of the enemy you kill, reasoned McNamara, the closer you are to winning. This was always a dubious idea, but the body count quickly became an informal metric for ranking units and handing out promotions, and was therefore often exaggerated. Counting bodies became a risky military objective in itself.
This episode symbolises the mindless, fruitless drive to count things. But it also shows us why metrics are so often used: McNamara was trying to understand and control a distant situation using the skills of a generalist, not a general. Muller shows that metrics are often used as a substitute for relevant experience, by managers with generic rather than specific expertise.
Muller does not claim that metrics are always useless, but that we expect too much from them as a tool of management. For example, if a group of doctors collect and analyse data on clinical outcomes, they are likely to learn something together. If bonuses and promotions are tied to the numbers, the exercise will teach nobody anything and may end up killing patients. Several studies have found evidence of cardiac surgeons refusing to operate on the sickest patients for fear of lowering their reported success rates.
It’s easy to sympathise with this argument, and I do. (I made some similar points in a chapter of my book Messy.) The Tyranny of Metrics does us a service in briskly pulling together parallel arguments from economics, management science, philosophy and psychology along with examples from education, policing, medicine, business and the military. It makes the case for professional autonomy: that metrics should be tools in the hands of teachers, doctors and soldiers rather than tools in the hands of those who would oversee them.
In an excellent final chapter, Muller summarises his argument thus: “measurement is not an alternative to judgement: measurement demands judgement: judgement about whether to measure, what to measure, how to evaluate the significance of what’s been measured, whether rewards and penalties will be attached to the results, and to whom to make the measurements available”.
This is a strong argument, but there are gaps in it. The book does not engage seriously enough with the possibility that the advantages of metric-driven accountability might outweigh the undoubted downsides. Tellingly, Muller complains of a university ratings metric that rewards high graduation rates, access for disadvantaged students, and low costs. He says these requirements are “mutually exclusive”, but they are not. They are in tension with each other, but a college that achieved all three goals would be a triumph rather than a logical absurdity.
The risk of trusting the professionals is that a bad teacher or police officer might coast undetected; or that a coterie of insiders might promote their own protégées, excluding women or ethnic minorities. Data-gathering efforts promise to spot prejudice, incompetence, back-scratching and worse. Perhaps they are doomed to fail, but in a world where insiders have covered up for each other far too often, we should not dismiss them too quickly.
Nor does this book reckon with evidence that mechanical statistical predictions often beat the subjective judgment of experts. This was demonstrated by psychologist Paul Meehl in his 1954 book Clinical versus Statistical Prediction. Subsequent research has supported his claim, while campaigners for evidence-based medicine such as Archie Cochrane and Sir Iain Chalmers have made a strong case not simply to take expert medical opinion on trust. Overconfident experts have been humbled by statistical methods frequently enough for the phenomenon to have been worth a chapter.
Finally, and perhaps most curiously, there is no discussion of computers, cheap sensors, or big data. In this respect, at least, the book could have been written in the 1980s. It is a strange omission, especially since Muller would surely have much to say about big data and algorithmic management.
All this, however, is criticism from a position of admiration. Many of us have the vague sense that metrics are leading us astray, stripping away context, devaluing subtle human judgement, and rewarding those who know how to play the system. Muller’s book crisply explains where this fashion came from, why it can be so counterproductive and why we don’t learn. It should be required reading for any manager on the verge of making the Vietnam body count mistake all over again.
Written for and first published in the Financial Times on 24 January 2018.