Odds and ends

13th October, 2012

Review by Tim Harford

An investigation into the art of forecasting argues for a pick-and-mix approach

The Signal and the Noise: The Art and Science of Prediction, by Nate Silver, Allen Lane RRP£25/Penguin Press, RRP$27.95, 544 pages

When it comes to soothsaying, there seem to be two types of person: those who will gladly and glibly opine on anything from the chance of rain tomorrow to the chance of Mitt Romney winning the presidency, and those who think the forecasting game is all but impossible, the exclusive preserve of fools and frauds.

This state of affairs is not good enough, says Nate Silver, a statistician most celebrated for his New York Times blog, FiveThirtyEight, and its forecasts of election results in the US. While prediction is indeed a difficult affair, it is not hopeless. Thoughtful people with serious theories and mathematical nous should get involved, argues Silver in The Signal and the Noise, not only because they have a good chance of raising the bar for forecasters but because prediction is the acid test of their expertise.

Although Silver is a numbers man, and his book is seasoned with graphs, tables and the occasional equation, he advocates a pick-and-mix approach to forecasting. A statistical model may or may not beat expert judgment, and a computer may or may not out-forecast a human, but a judicious combination of approaches will generally outperform any particular method.

What works in forecasting is not always effective when it comes to writing a book. Each chapter picks a different forecasting problem, from terrorism to baseball to climate change. These individual chapters are strong, and some of them are outstanding. The analysis of the subprime crisis is as lucid as any I have read, I was hooked by his account of computer chess and the explanation of weather forecasting is a revelation.

The whole is less than the sum of the parts, alas. A chapter on poker seems to be there because Silver was once a professional poker player; the chess chapter, while brilliant, seems barely relevant. His pen portraits do not satisfy (do we care about Robert Daum, whose research is briefly cited, now that we know he is “a doctor’s doctor, with a dignified voice, a beard, and a detached sense of humor”?) and at times his efforts to document his extensive research descend into name dropping. “I was told by Tim Berners-Lee,” he writes, of some fact or other. After the first 400 pages of this kind of thing one begins to wish that Silver’s editor had been more assertive.

Despite these frustrations, there is a great deal to admire in the book. It defies easy summary but at its heart is the admonition that we should all think more like Thomas Bayes, an 18th-century minister and mathematician, nonconformist in both roles. Bayes’ theorem, published posthumously, tells us how to combine our pre-existing view of the world with new information in a rational way.

Bayes’ theorem can produce some counterintuitive results. My colleague John Kay once published a pair of columns about the classic game show “Let’s Make A Deal”, in which a grand prize lurks in one of three boxes. The contestant provisionally chooses a box; the host of the show opens a different box to reveal no prize; then the contestant must decide whether to open her chosen box, or to switch at the last minute and open the only alternative. Bayes’ theorem demonstrates quite clearly that the contestant should switch, but Kay’s postbag was testimony to the fact that few people believe this conclusion.

Silver explains Bayes’ theorem with a dark example: the attacks on the World Trade Center. When the first plane hit the tower, horrified observers instinctively updated the possibility of a terrorist attack that day from “barely thinkable” to “distinctly possible”, although at that stage an accident could not be discounted. Bayes’ theorem shows that when the second plane hit, the chance of terrorism could be updated again, from “distinctly possible” to “all but certain”.

There is no need for a mathematical analysis to tell us that, but Silver argues convincingly that Bayes’ theorem is an important reality check on our efforts to forecast the future. How, for instance, should we reconcile a large body of theory and evidence predicting global warming with the fact that there has been no warming trend over the last decade or so? Sceptics react with glee, while true believers dismiss the new information.

A better response is to use Bayes’ theorem: the lack of recent warming is evidence against recent global warming predictions, but it is weak evidence. This is because there is enough variability in global temperatures to make such an outcome unsurprising. The new information should reduce our confidence in our models of global warming – but only a little.

The same approach can be used in anything from an economic forecast to a hand of poker, and while Bayes’ theorem can be a formal affair, Bayesian reasoning also works as a rule of thumb. We tend to either dismiss new evidence, or embrace it as though nothing else matters. Bayesians try to weigh both the old hypothesis and the new evidence in a sensible way. This is good advice, and less technical than it might sound.

Despite its flaws, The Signal and the Noise is a book worth reading. It says something new in a crowded field, it is fun to read, and it’s full of facts you will remember. There is some noise here, but Silver has also produced a signal that is a pleasure to follow.

Also published at ft.com.

Pin It on Pinterest

Share This