Tim Harford The Undercover Economist


From the geeks who took over poker to the nuclear safety experts who want to prevent the next banking meltdown, these are my favourite long-form articles.


Big Data: Are we making a big mistake?

Five years ago, a team of researchers from Google announced a remarkable achievement in one of the world’s top scientific journals, Nature. Without needing the results of a single medical check-up, they were nevertheless able to track the spread of influenza across the US. What’s more, they could do it more quickly than the Centers for Disease Control and Prevention (CDC). Google’s tracking had only a day’s delay, compared with the week or more it took for the CDC to assemble a picture based on reports from doctors’ surgeries. Google was faster because it was tracking the outbreak by finding a correlation between what people searched for online and whether they had flu symptoms.

Not only was “Google Flu Trends” quick, accurate and cheap, it was theory-free. Google’s engineers didn’t bother to develop a hypothesis about what search terms – “flu symptoms” or “pharmacies near me” – might be correlated with the spread of the disease itself. The Google team just took their top 50 million search terms and let the algorithms do the work.
The success of Google Flu Trends became emblematic of the hot new trend in business, technology and science: “Big Data”. What, excited journalists asked, can science learn from Google?
As with so many buzzwords, “big data” is a vague term, often thrown around by people with something to sell. Some emphasise the sheer scale of the data sets that now exist – the Large Hadron Collider’s computers, for example, store 15 petabytes a year of data, equivalent to about 15,000 years’ worth of your favourite music.
But the “big data” that interests many companies is what we might call “found data”, the digital exhaust of web searches, credit card payments and mobiles pinging the nearest phone mast. Google Flu Trends was built on found data and it’s this sort of data that ­interests me here. Such data sets can be even bigger than the LHC data – Facebook’s is – but just as noteworthy is the fact that they are cheap to collect relative to their size, they are a messy collage of datapoints collected for disparate purposes and they can be updated in real time. As our communication, leisure and commerce have moved to the internet and the internet has moved into our phones, our cars and even our glasses, life can be recorded and quantified in a way that would have been hard to imagine just a decade ago.
Cheerleaders for big data have made four exciting claims, each one reflected in the success of Google Flu Trends: that data analysis produces uncannily accurate results; that every single data point can be captured, making old statistical sampling techniques obsolete; that it is passé to fret about what causes what, because statistical correlation tells us what we need to know; and that scientific or statistical models aren’t needed because, to quote “The End of Theory”, a provocative essay published in Wired in 2008, “with enough data, the numbers speak for themselves”.
Unfortunately, these four articles of faith are at best optimistic oversimplifications. At worst, according to David Spiegelhalter, Winton Professor of the Public Understanding of Risk at Cambridge university, they can be “complete bollocks. Absolute nonsense.”
Found data underpin the new internet economy as companies such as Google, Facebook and Amazon seek new ways to understand our lives through our data exhaust. Since Edward Snowden’s leaks about the scale and scope of US electronic surveillance it has become apparent that security services are just as fascinated with what they might learn from our data exhaust, too.
Consultants urge the data-naive to wise up to the potential of big data. A recent report from the McKinsey Global Institute reckoned that the US healthcare system could save $300bn a year – $1,000 per American – through better integration and analysis of the data produced by everything from clinical trials to health insurance transactions to smart running shoes.
But while big data promise much to scientists, entrepreneurs and governments, they are doomed to disappoint us if we ignore some very familiar statistical lessons.
“There are a lot of small data problems that occur in big data,” says Spiegelhalter. “They don’t disappear because you’ve got lots of the stuff. They get worse.”
. . .
Four years after the original Nature paper was published, Nature News had sad tidings to convey: the latest flu outbreak had claimed an unexpected victim: Google Flu Trends. After reliably providing a swift and accurate account of flu outbreaks for several winters, the theory-free, data-rich model had lost its nose for where flu was going. Google’s model pointed to a severe outbreak but when the slow-and-steady data from the CDC arrived, they showed that Google’s estimates of the spread of flu-like illnesses were overstated by almost a factor of two.
The problem was that Google did not know – could not begin to know – what linked the search terms with the spread of flu. Google’s engineers weren’t trying to figure out what caused what. They were merely finding statistical patterns in the data. They cared about ­correlation rather than causation. This is common in big data analysis. Figuring out what causes what is hard (impossible, some say). Figuring out what is correlated with what is much cheaper and easier. That is why, according to Viktor Mayer-Schönberger and Kenneth Cukier’s book, Big Data, “causality won’t be discarded, but it is being knocked off its pedestal as the primary fountain of meaning”.
But a theory-free analysis of mere correlations is inevitably fragile. If you have no idea what is behind a correlation, you have no idea what might cause that correlation to break down. One explanation of the Flu Trends failure is that the news was full of scary stories about flu in December 2012 and that these stories provoked internet searches by people who were healthy. Another possible explanation is that Google’s own search algorithm moved the goalposts when it began automatically suggesting diagnoses when people entered medical symptoms.
Google Flu Trends will bounce back, recalibrated with fresh data – and rightly so. There are many reasons to be excited about the broader opportunities offered to us by the ease with which we can gather and analyse vast data sets. But unless we learn the lessons of this episode, we will find ourselves repeating it.
Statisticians have spent the past 200 years figuring out what traps lie in wait when we try to understand the world through data. The data are bigger, faster and cheaper these days – but we must not pretend that the traps have all been made safe. They have not.
. . .
In 1936, the Republican Alfred Landon stood for election against President Franklin Delano Roosevelt. The respected magazine, The Literary Digest, shouldered the responsibility of forecasting the result. It conducted a postal opinion poll of astonishing ambition, with the aim of reaching 10 million people, a quarter of the electorate. The deluge of mailed-in replies can hardly be imagined but the Digest seemed to be relishing the scale of the task. In late August it reported, “Next week, the first answers from these ten million will begin the incoming tide of marked ballots, to be triple-checked, verified, five-times cross-classified and totalled.”
After tabulating an astonishing 2.4 million returns as they flowed in over two months, The Literary Digest announced its conclusions: Landon would win by a convincing 55 per cent to 41 per cent, with a few voters favouring a third candidate.
The election delivered a very different result: Roosevelt crushed Landon by 61 per cent to 37 per cent. To add to The Literary Digest’s agony, a far smaller survey conducted by the opinion poll pioneer George Gallup came much closer to the final vote, forecasting a comfortable victory for Roosevelt. Mr Gallup understood something that The Literary Digest did not. When it comes to data, size isn’t everything.
Opinion polls are based on samples of the voting population at large. This means that opinion pollsters need to deal with two issues: sample error and sample bias.
Sample error reflects the risk that, purely by chance, a randomly chosen sample of opinions does not reflect the true views of the population. The “margin of error” reported in opinion polls reflects this risk and the larger the sample, the smaller the margin of error. A thousand interviews is a large enough sample for many purposes and Mr Gallup is reported to have conducted 3,000 interviews.
But if 3,000 interviews were good, why weren’t 2.4 million far better? The answer is that sampling error has a far more dangerous friend: sampling bias. Sampling error is when a randomly chosen sample doesn’t reflect the underlying population purely by chance; sampling bias is when the sample isn’t randomly chosen at all. George Gallup took pains to find an unbiased sample because he knew that was far more important than finding a big one.
The Literary Digest, in its quest for a bigger data set, fumbled the question of a biased sample. It mailed out forms to people on a list it had compiled from automobile registrations and telephone directories – a sample that, at least in 1936, was disproportionately prosperous. To compound the problem, Landon supporters turned out to be more likely to mail back their answers. The combination of those two biases was enough to doom The Literary Digest’s poll. For each person George Gallup’s pollsters interviewed, The Literary Digest received 800 responses. All that gave them for their pains was a very precise estimate of the wrong answer.
The big data craze threatens to be The Literary Digest all over again. Because found data sets are so messy, it can be hard to figure out what biases lurk inside them – and because they are so large, some analysts seem to have decided the sampling problem isn’t worth worrying about. It is.
Professor Viktor Mayer-Schönberger of Oxford’s Internet Institute, co-author of Big Data, told me that his favoured definition of a big data set is one where “N = All” – where we no longer have to sample, but we have the entire background population. Returning officers do not estimate an election result with a representative tally: they count the votes – all the votes. And when “N = All” there is indeed no issue of sampling bias because the sample includes everyone.
But is “N = All” really a good description of most of the found data sets we are considering? Probably not. “I would challenge the notion that one could ever have all the data,” says Patrick Wolfe, a computer scientist and professor of statistics at University College London.
An example is Twitter. It is in principle possible to record and analyse every message on Twitter and use it to draw conclusions about the public mood. (In practice, most researchers use a subset of that vast “fire hose” of data.) But while we can look at all the tweets, Twitter users are not representative of the population as a whole. (According to the Pew Research Internet Project, in 2013, US-based Twitter users were disproportionately young, urban or suburban, and black.)
There must always be a question about who and what is missing, especially with a messy pile of found data. Kaiser Fung, a data analyst and author of Numbersense, warns against simply assuming we have everything that matters. “N = All is often an assumption rather than a fact about the data,” he says.
Consider Boston’s Street Bump smartphone app, which uses a phone’s accelerometer to detect potholes without the need for city workers to patrol the streets. As citizens of Boston download the app and drive around, their phones automatically notify City Hall of the need to repair the road surface. Solving the technical challenges involved has produced, rather beautifully, an informative data exhaust that addresses a problem in a way that would have been inconceivable a few years ago. The City of Boston proudly proclaims that the “data provides the City with real-time in­formation it uses to fix problems and plan long term investments.”
Yet what Street Bump really produces, left to its own devices, is a map of potholes that systematically favours young, affluent areas where more people own smartphones. Street Bump offers us “N = All” in the sense that every bump from every enabled phone can be recorded. That is not the same thing as recording every pothole. As Microsoft researcher Kate Crawford points out, found data contain systematic biases and it takes careful thought to spot and correct for those biases. Big data sets can seem comprehensive but the “N = All” is often a seductive illusion.
. . .
Who cares about causation or sampling bias, though, when there is money to be made? Corporations around the world must be salivating as they contemplate the uncanny success of the US discount department store Target, as famously reported by Charles Duhigg in The New York Times in 2012. Duhigg explained that Target has collected so much data on its customers, and is so skilled at analysing that data, that its insight into consumers can seem like magic.
Duhigg’s killer anecdote was of the man who stormed into a Target near Minneapolis and complained to the manager that the company was sending coupons for baby clothes and maternity wear to his teenage daughter. The manager apologised profusely and later called to apologise again – only to be told that the teenager was indeed pregnant. Her father hadn’t realised. Target, after analysing her purchases of unscented wipes and magnesium supplements, had.
Statistical sorcery? There is a more mundane explanation.
“There’s a huge false positive issue,” says Kaiser Fung, who has spent years developing similar approaches for retailers and advertisers. What Fung means is that we didn’t get to hear the countless stories about all the women who received coupons for babywear but who weren’t pregnant.
Hearing the anecdote, it’s easy to assume that Target’s algorithms are infallible – that everybody receiving coupons for onesies and wet wipes is pregnant. This is vanishingly unlikely. Indeed, it could be that pregnant women receive such offers merely because everybody on Target’s mailing list receives such offers. We should not buy the idea that Target employs mind-readers before considering how many misses attend each hit.
In Charles Duhigg’s account, Target mixes in random offers, such as coupons for wine glasses, because pregnant customers would feel spooked if they realised how intimately the company’s computers understood them.
Fung has another explanation: Target mixes up its offers not because it would be weird to send an all-baby coupon-book to a woman who was pregnant but because the company knows that many of those coupon books will be sent to women who aren’t pregnant after all.
None of this suggests that such data analysis is worthless: it may be highly profitable. Even a modest increase in the accuracy of targeted special offers would be a prize worth winning. But profitability should not be conflated with omniscience.
. . .
In 2005, John Ioannidis, an epidemiologist, published a research paper with the self-explanatory title, “Why Most Published Research Findings Are False”. The paper became famous as a provocative diagnosis of a serious issue. One of the key ideas behind Ioannidis’s work is what statisticians call the “multiple-comparisons problem”.
It is routine, when examining a pattern in data, to ask whether such a pattern might have emerged by chance. If it is unlikely that the observed pattern could have emerged at random, we call that pattern “statistically significant”.
The multiple-comparisons problem arises when a researcher looks at many possible patterns. Consider a randomised trial in which vitamins are given to some primary schoolchildren and placebos are given to others. Do the vitamins work? That all depends on what we mean by “work”. The researchers could look at the children’s height, weight, prevalence of tooth decay, classroom behaviour, test scores, even (after waiting) prison record or earnings at the age of 25. Then there are combinations to check: do the vitamins have an effect on the poorer kids, the richer kids, the boys, the girls? Test enough different correlations and fluke results will drown out the real discoveries.
There are various ways to deal with this but the problem is more serious in large data sets, because there are vastly more possible comparisons than there are data points to compare. Without careful analysis, the ratio of genuine patterns to spurious patterns – of signal to noise – quickly tends to zero.
Worse still, one of the antidotes to the ­multiple-comparisons problem is transparency, allowing other researchers to figure out how many hypotheses were tested and how many contrary results are languishing in desk drawers because they just didn’t seem interesting enough to publish. Yet found data sets are rarely transparent. Amazon and Google, Facebook and Twitter, Target and Tesco – these companies aren’t about to share their data with you or anyone else.
New, large, cheap data sets and powerful ­analytical tools will pay dividends – nobody doubts that. And there are a few cases in which analysis of very large data sets has worked miracles. David Spiegelhalter of Cambridge points to Google Translate, which operates by statistically analysing hundreds of millions of documents that have been translated by humans and looking for patterns it can copy. This is an example of what computer scientists call “machine learning”, and it can deliver astonishing results with no preprogrammed grammatical rules. Google Translate is as close to theory-free, data-driven algorithmic black box as we have – and it is, says Spiegelhalter, “an amazing achievement”. That achievement is built on the clever processing of enormous data sets.
But big data do not solve the problem that has obsessed statisticians and scientists for centuries: the problem of insight, of inferring what is going on, and figuring out how we might intervene to change a system for the better.
“We have a new resource here,” says Professor David Hand of Imperial College London. “But nobody wants ‘data’. What they want are the answers.”
To use big data to produce such answers will require large strides in statistical methods.
“It’s the wild west right now,” says Patrick Wolfe of UCL. “People who are clever and driven will twist and turn and use every tool to get sense out of these data sets, and that’s cool. But we’re flying a little bit blind at the moment.”
Statisticians are scrambling to develop new methods to seize the opportunity of big data. Such new methods are essential but they will work by building on the old statistical lessons, not by ignoring them.
Recall big data’s four articles of faith. Uncanny accuracy is easy to overrate if we simply ignore false positives, as with Target’s pregnancy predictor. The claim that causation has been “knocked off its pedestal” is fine if we are making predictions in a stable environment but not if the world is changing (as with Flu Trends) or if we ourselves hope to change it. The promise that “N = All”, and therefore that sampling bias does not matter, is simply not true in most cases that count. As for the idea that “with enough data, the numbers speak for themselves” – that seems hopelessly naive in data sets where spurious patterns vastly outnumber genuine discoveries.
“Big data” has arrived, but big insights have not. The challenge now is to solve new problems and gain new answers – without making the same old statistical mistakes on a grander scale than ever.

This article was first published in the FT Magazine, 29/30 March 2014. Read it in its original setting here.


What next for behavioural economics?

The past decade has been a triumph for behavioural economics, the fashionable cross-breed of psychology and economics. First there was the award in 2002 of the Nobel Memorial Prize in economics to a psychologist, Daniel Kahneman – the man who did as much as anything to create the field of behavioural economics. Bestselling books were launched, most notably by Kahneman himself (Thinking, Fast and Slow , 2011) and by his friend Richard Thaler, co-author of Nudge (2008). Behavioural economics seems far sexier than the ordinary sort, too: when last year’s Nobel was shared three ways, it was the behavioural economist Robert Shiller who grabbed all the headlines.
Behavioural economics is one of the hottest ideas in public policy. The UK government’s Behavioural Insights Team (BIT) uses the discipline to craft better policies, and in February was part-privatised with a mission to advise governments around the world. The White House announced its own behavioural insights team last summer.
So popular is the field that behavioural economics is now often misapplied as a catch-all term to refer to almost anything that’s cool in popular social science, from the storycraft of Malcolm Gladwell, author of The Tipping Point (2000), to the empirical investigations of Steven Levitt, co-author of Freakonomics (2005).
Yet, as with any success story, the backlash has begun. Critics argue that the field is overhyped, trivial, unreliable, a smokescreen for bad policy, an intellectual dead-end – or possibly all of the above. Is behavioural economics doomed to reflect the limitations of its intellectual parents, psychology and economics? Or can it build on their strengths and offer a powerful set of tools for policy makers and academics alike?
A recent experiment designed by BIT highlights both the opportunity and the limitations of the new discipline. The trial was designed to encourage people to sign up for the Organ Donor Register. It was huge; more than a million people using the Driver and Vehicle Licensing Agency website were shown a webpage inviting them to become an organ donor. One of eight different messages was displayed at random. One was minimalist, another spoke of the number of people who die while awaiting donations, yet another appealed to the idea of reciprocity – if you needed an organ, wouldn’t you want someone to donate an organ to you?
BIT devoted particular attention to an idea called “social proof”, made famous 30 years ago by psychologist Robert Cialdini’s book Influence. While one might be tempted to say, “Too few people are donating their organs, we desperately need your help to change that”, the theory of social proof says that’s precisely the wrong thing to do. Instead, the persuasive message will suggest: “Every day, thousands of people sign up to be donors, please join them.” Social proof describes our tendency to run with the herd; why else are books marketed as “bestsellers”?
Expecting social proof to be effective, the BIT trial used three different variants of a social proof message, one with a logo, one with a photo of smiling people, and one unadorned. None of these approaches was as successful as the best alternatives at persuading people to sign up as donors. The message with the photograph – for which the teams had high hopes – was a flop, proving worse than no attempt at persuasion at all.
Three points should be made here. The first is that this is exactly why running trials is an excellent idea: had the rival approaches not been tested with an experiment, it would have been easy for well-meaning civil servants acting on authoritative advice to have done serious harm. The trial was inexpensive, and now that the most persuasive message is in use (“If you needed an organ transplant, would you have one? If so, please help others”), roughly 100,000 additional people can be expected to sign up for the donor register each year.
The second point is that there is something unnerving about a discipline in which our discoveries about the past do not easily generalise to the future. Social proof is a widely accepted idea in psychology but, as the donor experiment shows, it does not always apply and it can be hard to predict when or why.
This patchwork of sometimes-fragile psychological results hardly invalidates the whole field but complicates the business of making practical policy. There is a sense that behavioural economics is just regular economics plus common sense – but since psychology isn’t mere common sense either, applying psychological lessons to economics is not a simple task.
The third point is that the organ donor experiment has little or nothing to do with behavioural economics, strictly defined. “The Behavioural Insights Team is widely perceived as doing behavioural economics,” says Daniel Kahneman. “They are actually doing social psychology.”
. . .
The line between behavioural economics and psychology can get a little blurred. Behavioural economics is based on the traditional “neoclassical” model of human behaviour used by economists. This essentially mathematical model says human decisions can usefully be modelled as though our choices were the outcome of solving differential equations. Add psychology into the mix – for example, Kahneman’s insight (with the late Amos Tversky) that we treat the possibility of a loss differently from the way we treat the possibility of a gain – and the task of the behavioural economist is to incorporate such ideas without losing the mathematically-solvable nature of the model.
Why bother with the maths? Consider the example of, say, improving energy efficiency. A psychologist might point out that consumers are impatient, poorly-informed and easily swayed by what their neighbours are doing. It’s the job of the behavioural economist to work out how energy markets might work under such conditions, and what effects we might expect if we introduced policies such as a tax on domestic heating or a subsidy for insulation.
It’s this desire to throw out the hyper-rational bathwater yet keep the mathematically tractable baby that leads to difficult compromises, and not everyone is happy. Economic traditionalists argue that behavioural economics is now hopelessly patched-together; some psychologists claim it’s still attempting to be too systematic.
Nick Chater, a psychologist at Warwick Business School and an adviser to the BIT, is a sympathetic critic of the behavioural economics approach. “The brain is the most rational thing in the universe”, he says, “but the way it solves problems is ad hoc and very local.” That suggests that attempts to formulate general laws of human behaviour may never be more than a rough guide to policy.
This shift to radical incrementalism is so much more important than some of the grand proposals out there
The most well-known critique of behavioural economics comes from a psychologist, Gerd Gigerenzer of the Max Planck Institute for Human Development. Gigerenzer argues that it is pointless to keep adding frills to a mathematical account of human behaviour that, in the end, has nothing to do with real cognitive processes.
I put this critique to David Laibson, a behavioural economist at Harvard University. He concedes that Gigerenzer has a point but adds: “Gerd’s models of heuristic decision-making are great in the specific domains for which they are designed but they are not general models of behaviour.” In other words, you’re not going to be able to use them to figure out how people should, or do, budget for Christmas or nurse their credit card limit through a spell of joblessness.
Richard Thaler of the University of Chicago, who with Kahneman and Tversky is the founding father of behavioural economics, agrees. To discard the basic neoclassical framework of economics means “throwing away a lot of stuff that’s useful”.
For some economists, though, behavioural economics has already conceded too much to the patchwork of psychology. David K Levine, an economist at Washington University in St Louis, and author of Is Behavioral Economics Doomed? (2012), says: “There is a tendency to propose some new theory to explain each new fact. The world doesn’t need a thousand different theories to explain a thousand different facts. At some point there needs to be a discipline of trying to explain many facts with one theory.”
The challenge for behavioural economics is to elaborate on the neoclassical model to deliver psychological realism without collapsing into a mess of special cases. Some say that the most successful special case comes from Harvard’s David Laibson. It is a mathematical tweak designed to represent the particular brand of short-termism that leads us to sign up for the gym yet somehow never quite get around to exercising. It’s called “hyperbolic discounting”, a name that refers to a mathematical curve, and which says much about the way behavioural economists represent human psychology.
The question is, how many special cases can behavioural economics sustain before it becomes arbitrary and unwieldy? Not more than one or two at a time, says Kahneman. “You might be able to do it with two but certainly not with many factors.” Like Kahneman, Thaler believes that a small number of made-for-purpose behavioural economics models have proved their worth already. He argues that trying to unify every psychological idea in a single model is pointless. “I’ve always said that if you want one unifying theory of economic behaviour, you won’t do better than the neoclassical model, which is not particularly good at describing actual decision making.”
. . .
Meanwhile, the policy wonks plug away at the rather different challenge of running rigorous experiments with public policy. There is something faintly unsatisfying about how these policy trials have often confirmed what should have been obvious. One trial, for example, showed that text message reminders increase the proportion of people who pay legal fines. This saves everyone the trouble of calling in the bailiffs. Other trials have shown that clearly-written letters with bullet-point summaries provoke higher response rates.
None of this requires the sophistication of a mathematical model of hyperbolic discounting or loss aversion. It is obvious stuff. Unfortunately it is obvious stuff that is often neglected by the civil service. It is hard to object to inexpensive trials that demonstrate a better way. Nick Chater calls the idea “a complete no-brainer”, while Kahneman says “you can get modest gains at essentially zero cost”.
David Halpern, a Downing Street adviser under Tony Blair, was appointed by the UK coalition government in 2010 to establish the BIT. He says that the idea of running randomised trials in government has now picked up steam. The Financial Conduct Authority has also used randomisation to develop more effective letters to people who may have been missold financial products. “This shift to radical incrementalism is so much more important than some of the grand proposals out there,” says Halpern.
Not everyone agrees. In 2010, behavioural economists George Loewenstein and Peter Ubel wrote in The New York Times that “behavioural economics is being used as a political expedient, allowing policy makers to avoid painful but more effective solutions rooted in traditional economics.”
For example, in May 2010, just before David Cameron came to power, he sang the praises of behavioural economics in a TED talk. “The best way to get someone to cut their electricity bill,” he said, “is to show them their own spending, to show them what their neighbours are spending, and then show what an energy-conscious neighbour is spending.”
But Cameron was mistaken. The single best way to promote energy efficiency is, almost certainly, to raise the price of energy. A carbon tax would be even better, because it not only encourages people to save energy but to switch to lower-carbon sources of energy. The appeal of a behavioural approach is not that it is more effective but that it is less unpopular.
Thaler points to the experience of Cass Sunstein, his Nudge co-author, who spent four years as regulatory tsar in the Obama White House. “Cass wanted a tax on petrol but he couldn’t get one, so he pushed for higher fuel economy standards. We all know that’s not as efficient as raising the tax on petrol – but that would be lucky to get a single positive vote in Congress.”
Should we be trying for something more ambitious than behavioural economics? “I don’t know if we know enough yet to be more ambitious,” says Kahneman, “But the knowledge that currently exists in psychology is being put to very good use.”
Small steps have taken behavioural economics a long way, says Laibson, citing savings policy in the US. “Every dimension of that environment is now behaviourally tweaked.” The UK has followed suit, with the new auto-enrolment pensions, directly inspired by Thaler’s work.
Laibson says behavioural economics has only just begun to extend its influence over public policy. “The glass is only five per cent full but there’s no reason to believe the glass isn’t going to completely fill up.”

First published on FT.com, Life and Arts, 22 March 2014

Free email updates

(You can unsubscribe at any time)


Do You Believe in Sharing?

While delivering his Nobel lecture in 2007, Al Gore declared: “Today, we dumped another 70 million tons of global-warming pollution into the thin shell of atmosphere surrounding our planet, as if it were an open sewer.”

It’s a powerful example of the way we tend to argue about the impact of the human race on the planet that supports us: statistical or scientific claims combined with a call to action. But the argument misses something important: if we are to act, then how? Who must do what, who will benefit and how will all this be agreed and policed?

To ask how people work together to deal with environmental problems is to ask one of the fundamental questions in social science: how do people work together at all? This is the story of two researchers who attacked the question in very different ways – and with very different results.

“The Tragedy of the Commons” is a seminal article about why some environmental problems are so hard to solve. It was published in the journal Science in 1968 and its influence was huge. Partly this was the zeitgeist: the late 1960s and early 1970s was an era of big environmental legislation and regulation in the US. Yet that cannot be the only reason that the “tragedy of the commons” has joined a very small group of concepts – such as the “prisoner’s dilemma” or the “selfish gene” – to have escaped from academia to take on a life of their own.

The credit must go to Garrett Hardin, the man who coined the phrase and wrote the article. Hardin was a respected ecologist but “The Tragedy of the Commons” wasn’t an ecological study. It wasn’t really a piece of original research at all.

“Nothing he wrote in there had not been said by fisheries economists,” says Daniel Cole, a professor at Indiana University and a scholar of Hardin’s research. The key idea, indeed, goes back to Aristotle. Hardin’s genius was in developing a powerful, succinct story with a memorable name.

The story goes as follows: imagine common pasture, land owned by everyone and no one, “open to all” for grazing livestock. Now consider the incentives faced by people bringing animals to feed. Each new cow brought to the pasture represents pure private profit for the individual herdsman in question. But the commons cannot sustain an infinite number of cows. At some stage it will be overgrazed and the ecosystem may fail. That risk is not borne by any individual, however, but by society as a whole.

With a little mathematical elaboration Hardin showed that these incentives led inescapably to ecological disaster and the collapse of the commons. The idea of a communally owned resource might be appealing but it was ultimately self-defeating.

It was in this context that Hardin deployed the word “tragedy”. He didn’t use it to suggest that this was sad. He meant that this was inevitable. Hardin, who argued that much of the natural sciences was grounded by limits – such as the speed of light or the force of gravity – quoted the philosopher Alfred North Whitehead, who wrote that tragedy “resides in the solemnity of the remorseless working of things”.

. . .

Lin Ostrom never believed in “the remorseless working of things”. Born Elinor Awan in Los Angeles in 1933, by the time she first saw Garrett Hardin present his ideas she had already beaten the odds.

Lin was brought up in Depression-era poverty after her Jewish father left her Protestant mother. She was bullied at school – Beverly Hills High, of all places – because she was half-Jewish. She divorced her first husband, Charles Scott, after he discouraged her from pursuing an academic career, where she suffered discrimination for years. Initially steered away from mathematics at school, Lin was rejected by the economics programme at UCLA. She was only – finally – accepted on a PhD in political science after observing that UCLA’s political science department hadn’t admitted a woman for 40 years.

She persevered and secured her PhD after studying the management of fresh water in Los Angeles. In the first half of the 20th century, the city’s water supply had been blighted by competing demands to pump fresh water for drinking and farming. By the 1940s, however, the conflicting parties had begun to resolve their differences. In both her PhD, which she completed in 1965, and subsequent research, Lin showed that such outcomes often came from private individuals or local associations, who came up with their own rules and then lobbied the state to enforce them. In the case of the Los Angeles water producers, they drew up contracts to share their resources and the city’s water supply stabilised.

It was only when Lin saw Hardin lecture that she realised that she had been studying the tragedy of the commons all along. It was 1968, the year that the famous article was published. Garrett Hardin was 53, in the early stages of a career as a campaigning public intellectual that would last the rest of his life. Lin was 35, now Ostrom: she had married Vincent Ostrom, a respected political scientist closer to Hardin’s age, and together they had moved to Indiana University. Watching Hardin lecture galvanised her. But that wasn’t because she was convinced he was right. It was because she was convinced that he was wrong.

In his essay, Hardin explained that there was no way to manage communal property sustainably. The only solution was to obliterate the communal aspect. Either the commons could be nationalised and managed by the state – a Leviathan for the age of environmentalism – or the commons could be privatised, divided up into little parcels and handed out to individual farmers, who would then look after their own land responsibly. The theory behind all this is impeccable and, despite coming from a biologist, highly appealing to anyone with an economics training.

But Lin Ostrom could see that there must be something wrong with the logic. Her research on managing water in Los Angeles, watching hundreds of different actors hammer out their messy yet functional agreements, provided a powerful counter-example to Hardin. She knew of other examples, too, in which common resources had been managed sustainably without Hardin’s black-or-white solutions.

The problem with Hardin’s logic was the very first step: the assumption that communally owned land was a free-for-all. It wasn’t. The commons were owned by a community. They were managed by a community. These people were neighbours. They lived next door to each other. In many cases, they set their own rules and policed those rules.

This is not to deny the existence of the tragedy of the commons altogether. Hardin’s analysis looks prescient when applied to our habit of pumping carbon dioxide into the atmosphere or overfishing the oceans. But the existence of clear counter-examples should make us hesitate before accepting Hardin’s argument that tragedy is unstoppable. Lin Ostrom knew that there was nothing inevitable about the self-destruction of “common pool resources”, as economists call them. The tragedy of the commons wasn’t a tragedy at all. It was a problem – and problems have solutions.

If Garrett Hardin and Lin Ostrom had reached different conclusions about the commons, perhaps that was because their entire approaches to academic research were different. Hardin wanted to change the world; Ostrom merely wanted to describe it.

That goal of description, though, was a vast project. Common pool resources could be found all over the planet, from the high meadows of Switzerland to the lobster fisheries of Maine, from forests in Sri Lanka to water in Nepal. Hardin’s article had sliced through the complexity with his assumption that all commons were in some sense the same. But they aren’t.

To describe even a single case study of governing a common resource is a challenge (Lin’s PhD was devoted to the West Basin water district of Los Angeles). Vincent Ostrom, Lin’s husband, had developed the idea of “polycentricity” in political science: polycentric systems have multiple, independent and overlapping sources of power and authority.

By their very nature, they are messy to describe and hard to compare with each other. Unfortunately for any tidy-minded social scientist, they are also everywhere.

Complicating the problem further was the narrow focus of academic specialities. Lin was encouraged that many people had been drawn, like her, to the study of common pool resources. But they were divided by discipline, by region and by subject: the sociologists didn’t talk to the economists; the India specialists didn’t talk to the Africanists; and the fishery experts didn’t know anything about forestry. As Ostrom and her colleagues at the University of Indiana looked into the problem they discovered more than a thousand separate case studies, each sitting in isolation.

Undeterred, they began to catalogue them, seeking to explain the difference between the successful attempts to manage environmental resources and the failures. There were the Swiss farmers of the village of Törbel, who had a system of rules, fines and local associations that dated from the 13th century to govern the use of scarce Alpine pastures and firewood. There were the fishermen of Alanya, in Turkey, who took part in a lottery each September to allocate fishing rights for the year ahead.

Over time, Ostrom developed a set of what she called “design principles” for managing common resources, drawn from what worked in the real world. She used the phrase hesitantly since, she argued, these arrangements were rarely designed or imposed from the top down; they usually evolved from the bottom up.

These principles included effective monitoring; graduated sanctions for those who break rules; and cheap access to conflict-resolution mechanisms (the fishermen of Alanya resolved their disputes in the local coffee house). There are several others. Ostrom wanted to be as precise as she could, to move away from the hand-waving of some social scientists. But there were limits to how reductive it was possible to be about such varied institutions. Lin’s only golden rule about common pool resources was that there are no panaceas.

Her work required a new set of intellectual tools. But for Ostrom, this effort was central to her academic life because knowledge itself – when you thought about it – was a kind of common pool resource as well. It could be squandered or it could be harvested for the public good. And it would only be harvested with the right set of rules.

Ostrom’s research project came to resemble one of the local, community-led institutions that she sought to explain. In 1973, the Ostroms established something called the “Workshop in Political Theory and Policy Analysis”. Why not a school or a centre or a department? It was partly to sidestep bureaucracy. “The university didn’t know what a workshop was,” says Michael McGinnis, a professor of political science at Indiana University and a colleague of the Ostroms. “They didn’t have rules for a workshop.”

But there was more behind the name than administrative guile. Vincent and Lin believed that the work they did was a kind of craft. (The couple had built their own home and made much of their own furniture, under the guidance of a local craftsman – the experience made an impression.) The students who attended didn’t call themselves students or researchers. They called themselves “workshoppers”.

The workshop under the Ostroms seems to have been a remarkable place, brightened up by Lin’s sparkling laugh and garish tops. (The laugh was a reliable sign that she was in the building, available to be buttonholed by students.) At reunions, Ostrom would lead the singing of folk songs; it was that kind of place. The Ostroms never had children but the workshoppers did – and those children called Lin “Grandma”.

. . .

The logic of Garrett Hardin’s 1968 essay is seductive but to read the text itself is a shock. Hardin’s policy proposals are extreme. He believed that the ultimate tragedy of the commons was overpopulation – and the central policy conclusion of the article was, to quote Hardin, that “freedom to breed is intolerable”.

In a 1974 essay, “Living on a Lifeboat”, he argued that it was pointless sending aid to starving people in Ethiopia. That would only make the real problem worse – the real problem being, of course, overpopulation.

Hardin robustly defended his views. In a 1987 interview with The New York Times, he opined, “There’s nothing more dangerous than a shallow-thinking compassionate person.

God, he can cause a lot of trouble.” But perhaps it was Hardin who was the one failing to think deeply enough. The logic of “The Tragedy of the Commons” worked well to frame a class of environmental problems. The danger was when Hardin leapt to drastic conclusions without looking at how other, similar-looking problems were being solved, again and again, by communities all over the world.

Nor has Hardin’s needle-sharp focus on overpopulation stood the test of time. When he published “The Tragedy of the Commons” in 1968, the growth rate of world population was higher than it had ever been – a rate at which population would double every 30 years. No wonder Hardin was alarmed. But birth rates have fallen dramatically. The world continues to face some severe environmental problems. However, it’s far from clear that “freedom to breed” is one of them.

There was no great public showdown between Lin Ostrom and Garrett Hardin, but Hardin did return to speak at Indiana University in 1976. The Ostroms invited him and some graduate students to dinner. Barbara Allen, now a professor at Carleton College, was one of them. She recalls that “the conversation was vigorous” as Hardin laid out his ideas for government-led initiatives to reduce the birth rate in the US, while Lin and Vincent worried about the unintended consequences of such top-down panaceas.

Allen recalls two other details: the way that Lin made space for her students to enter the argument and her joy in a new kitchen gadget she was using to make hamburgers for everyone. She loved “the odd delights of everyday life”, Allen later wrote, and loved to celebrate what worked.

Hardin, by contrast, seems to have been more of a pessimist about technology. “Technology does solve problems,” he told an interviewer in 1990, “but always at a cost.”

Lin Ostrom was a more optimistic character altogether. When she won the Nobel memorial prize for economics in 2009, she was the first woman to do so. She was quick to comment: “I won’t be the last.”

Some of her most recent research addressed the problem of climate change. Scientifically speaking, greenhouse gas emissions are a global pollutant, and so efforts have focused on establishing global agreements. That, said Ostrom, is a mistake. Common pool problems were usually too complex to solve from the top down; a polycentric approach was necessary, with people developing ideas and enforcing behaviour at a community, city, national and regional level.

Ostrom barely slowed down when she was diagnosed with pancreatic cancer in 2011. She kept going until the final days, leaving voicemail messages for Vincent who, at the age of 90, was deaf and beginning to become confused. (Her students would type them up and print them out in large fonts for him to read.) When Lin died last June, at the age of 78, she was reviewing a student’s PhD thesis. She’d been annotating the text, which lay on the table beside her hospital bed. Vincent died two weeks later. The couple left almost everything to the workshop.

Garrett Hardin and his wife Jane also died together, in September 2003. After 62 years of marriage, and both suffering from very poor health, they killed themselves. Perhaps strangely for a man who thought overpopulation was the world’s ultimate problem, Garrett Hardin had four children. But there may be a certain kind of logic in that. Hardin always felt that overpopulation was inevitable. He died the way he lived – a resolute believer in the remorseless working of things.

First published in the FT Magazine.


How the rich are making sure they stay on top

When the world’s richest countries were booming, few people worried overmuch that the top 1 per cent were enjoying an ever-growing share of that prosperity. In the wake of a depression in the US, a fiscal chasm in the UK and an existential crisis in the eurozone – and the shaming of the world’s bankers – worrying about inequality is no longer the preserve of the far left.

There should be no doubt about the facts: the income share of the top 1 per cent has roughly doubled in the US since the early 1970s, and is now about 20 per cent. Much the same trend can be seen in Australia, Canada and the UK – although in each case the income share of the top 1 per cent is smaller. In France, Germany and Japan there seems to be no such trend. (The source is the World Top Incomes Database, summarised in the opening paper of a superb symposium in this summer’s Journal of Economic Perspectives.)

But should we care? There are two reasons we might: process and outcome. We might worry that the gains of the rich are ill-gotten: the result of the old-boy network, or fraud, or exploiting the largesse of the taxpayer. Or we might worry that the results are noxious: misery and envy, or ill-health, or dysfunctional democracy, or slow growth as the rich sit on their cash, or excessive debt and thus financial instability.

Following the crisis, it might be unfashionable to suggest that the rich actually earned their money. But knee-jerk banker-bashers should take a look at research by Steven Kaplan and Joshua Rauh, again in the JEP symposium. They simply compare the fate of the top earners across different lines of business. Worried that chief executives are filling their boots thanks to the weak governance of publicly listed companies? So am I, but partners in law firms are also doing very nicely, as are the bosses of privately owned companies, as are the managers of hedge funds, as are top sports stars. Governance arrangements in each case are different.

Perhaps, then, some broad social norm has shifted, allowing higher pay across the board? If so, we would expect publicly scrutinised salaries to be catching up with those who have more privacy – for instance, managers of privately held corporations. The reverse is the case.

The uncomfortable truth is that market forces – that is, the result of freely agreed contracts – are probably behind much of the rise in inequality. Globalisation and technological change favour the highly skilled. In the middle of the income distribution, a strong pair of arms, a willingness to work hard and a bit of common sense used to provide a comfortable income. No longer. Meanwhile at the very top, winner-take-all markets are emerging, where the best or luckiest entrepreneurs, fund managers, authors or athletes hoover up most of the gains. The idea that the fat cats simply stole everyone else’s cream is emotionally powerful; it is not entirely convincing.

In a well-functioning market, people only earn high incomes if they create enough economic value to justify those incomes. But even if we could be convinced that this was true, we do not have to let the matter drop.

This is partly because the sums involved are immense. Between 1993 and 2011, in the US, average incomes grew a modest 13.1 per cent in total. But the average income of the poorest 99 per cent – that is everyone up to families making about $370,000 a year – grew just 5.8 per cent. That gap is a measure of just how much the top 1 per cent are making. The stakes are high.

I set out two reasons why we might care about inequality: an unfair process or a harmful outcome. But what really should concern us is that the two reasons are not actually distinct after all. The harmful outcome and the unfair process feed each other. The more unequal a society becomes, the greater the incentive for the rich to pull up the ladder behind them.

At the very top of the scale, plutocrats can shape the conversation by buying up newspapers and television channels or funding political campaigns. The merely prosperous scramble desperately to get their children into the right neighbourhood, nursery, school, university and internship – we know how big the gap has grown between winners and also-rans.

Miles Corak, another contributor to the JEP debate, is an expert on intergenerational income mobility, the question of whether rich parents have rich children. The painful truth is that in the most unequal developed nations – the UK and the US – the intergenerational transmission of income is stronger. In more equal societies such as Denmark, the tendency of privilege to breed privilege is much lower.

This is what sticks in the throat about the rise in inequality: the knowledge that the more unequal our societies become, the more we all become prisoners of that inequality. The well-off feel that they must strain to prevent their children from slipping down the income ladder. The poor see the best schools, colleges, even art clubs and ballet classes, disappearing behind a wall of fees or unaffordable housing.

The idea of a free, market-based society is that everyone can reach his or her potential. Somewhere, we lost our way.

The Undercover Economist Strikes Back’ by Tim Harford is published this month in the UK and in January in the US.

First published in the Financial Times, 16 August 2012.


Cory Doctorow has Lunch with the FT

Illustration by James Ferguson of Cory Doctorow

Portrait by James Ferguson

Cory Doctorow should be too busy for lunch. He’s co-editor of, and a prolific contributor to, one of the most influential blogs in the world, Boing Boing. Over the past decade the Canadian-born writer has published 16 books, mostly science fiction novels. He campaigns vigorously on the politics of the digital age. His speaking schedule for the two months following our lunch requires three return trips from his home in east London to North America. He has almost 300,000 followers on Twitter. He is an impeccably prompt email correspondent.
More remarkable to me than any of this is that he claims to prepare for himself, his wife and five-year-old daughter “a three-to-four-course, hot/cold tailor-made breakfast every morning, in 20 minutes flat, with handmade coffees”. And although I arrive at Hawksmoor 10 minutes early, he’s there already, sipping sparkling water at the bar and reading a book. He’s wearing thick-rimmed spectacles worthy of Eric Morecambe, a Disney “Haunted Mansion” T-shirt, and a jacket; he’s 41 but looks younger. Did I mention that I have a tiny crush on Cory Doctorow?

As we’re shown to our table at the window, I feel compelled to ask about the breakfasts. How does he do it? After reading about this quotidian feat of fatherhood, I had tried to make my wife a fancy breakfast in bed, with eggs and honey-drizzled yoghurt and other trimmings. It took long enough that well before I was finished she had surfaced to investigate what was going on. I ask Doctorow for tips, confessing that I can scarcely produce a gourmet breakfast for my family once a fortnight.

“That’s your problem,” he says, in a bright, brisk Toronto accent. “You don’t do it often enough. If you did it every day, you’d get very good at it. It would become a habit, and habits are free.” His own breakfasts are prepared the night before – porridge measured out, yoghurt and berries mixed, served and in the fridge, eggs in the saucepan ready to be boiled. It’s obsessive, precise, carefully optimised – and, it seems, highly effective.
The waiter arrives to discuss steak with us. Hawksmoor is a hipsterish steakhouse near Spitalfields market, all dark wood and brick. The menu is unconventional, with steaks priced by the gramme and particular weights of pre-cut steak chalked up on the board. Doctorow – who had sent me a list of places he’d be happy to eat – seems to be a regular, and quizzes the waiter about when the meat is delivered and why, when he comes in the evenings, certain cuts have already been crossed off. The waiter assures us that the meat is delivered fresh every day and never frozen, even though it’s harder to carve the steak in an unfrozen state.
“Unless you have a laser,” offers Doctorow, at which point the waiter, rather curiously, begins to discuss whether certain cuts cooked rare present a risk of food poisoning.
“We’re not really supposed to talk about food poisoning,” admits the waiter. “You’ve got to come up with another name for food poisoning,” suggests Doctorow. “Like, er, ‘exotic gut flora experience’.”
I explain to Doctorow that he can choose whatever he likes and the FT will pay, but the world gets to see the bill. “That’s a very funny little bit of behavioural economics,” he replies.
Some of the steaks are sized for two, and I indicate that I’m willing to share one. Doctorow selects a large porterhouse for us, and we’re persuaded for reasons of flavour rather than safety to go for medium rare rather than rare. Doctorow chooses bone marrow and onion to start, with creamed spinach to accompany the steak. I start with a Caesar salad, and order triple-cooked chips. I press him to order some wine and he reluctantly agrees to drink half a glass, but refuses to choose.
I express surprise that he claims to know nothing about wine although he is obsessive about, for example, coffee. “I specialise,” he explains, adding that he rarely drinks much. I choose two glasses of the cheapest red. It’s not bad, and he downs his swiftly.
. . .
Doctorow’s fiction champions technology, while warning of how easily it can be used by repressive states or corporations. His own life provides an example of how to live with freedom in a technological age – he’s a man with no particular title, no hierarchical authority, no corner office and no secretary, who somehow manages to keep the plates spinning. Is it the same relentless, nerdy optimisation that gets those breakfasts on the table? He quotes from Brian Eno’s collection of not-quite-aphorisms Oblique Strategies , “Be the first to not do what nobody has ever thought of not doing before.”
Boing Boing, a marvellously eclectic blog, is a case in point – it’s a stripped-down vision of a 21st-century media outlet. Founded in 1988 as a print magazine, it went digital in the mid-1990s, and became one of the first blogs to attract a mass audience. Doctorow started writing for it as a favour for the blog’s founder Mark Frauenfelder, who was going on holiday, and never stopped. It’s incorporated as a business and is funded by sponsors, advertising and merchandise. It has a wide reach and yet, by the standards of a newspaper, is produced by a tiny team, with four main writers, three of whom live in California.
“We are spectacularly lean,” says Doctorow. “We have one phone call a year if we need it. We have one meeting a year.” He’s saving on air fare by tacking this year’s editorial meeting – in Los Angeles – on to a pre-existing trip. It’s usually at the Magic Castle, a private club for magicians. This is a typical touch of whimsy; Doctorow is also seriously smitten by Disney theme parks (his first novel, Down and Out in the Magic Kingdom (2003), imagines what Walt Disney World might be in the 22nd century).
Is this “spectacularly lean” operation the future of newspaper publishing, I ask? “It’s a future of publishing. One of the things that newspapers obscured was that they weren’t a medium, they were a collection of media bodged together.” Newspapers are like books, he says, a format that encompasses “anything from actuarial tables to Mein Kampf”.
But I haven’t quite tired of the topic of getting things done. Doctorow says he’s published 16 books in the past decade. How?
“I figure out how much time I have to write a book. I figure out how many words I need to write. I convert that into a daily rate and I write that many words every day come hell or high water.” Before I can raise the question of quality, he goes on to explain that there’s very little correlation with what he thinks is good writing while he’s at the keyboard, and what later turns out to be good writing – and so he might as well just get the words down and sort it all out later. Lest that process sound like pure hackwork, Doctorow novels have won or been nominated for most of the science-fiction awards that count.
The “write it now and fix it later” approach sounds perfectly reasonable to me, but then Doctorow pushes it to an extreme. “For instance, I wrote Homeland [2013] while I was touring Germany to publicise Little Brother [2008]. I had a translator, we’d visit lots of schools, and so I’d be speaking English half the time and he’d be speaking German half the time, and I’d write the book while he was speaking German.”
I point out that he is describing a superpower. Didn’t people wonder what he was doing as he sat in front of an audience tapping away on his computer while his translator spoke?
“Yes, but that was fine. At the end of the talk someone would say,” – and Doctorow assumes a gentle German accent – “‘Herr Doctorow, what are you doing with your computer on the stage?’ and I’d explain that I was writing my next book. They’d love that.”
Science fiction is often a way of exploring issues of contemporary relevance, and Doctorow’s work is no exception. In For the Win (2010), a novel aimed at the “young adult” market, he describes a battle between internationally mobile capital and the attempts of the trade union movement to mobilise “virtual sweatshop” workers across international boundaries. The action moves between India, where anything goes in a deregulated environment, and China, where the state is powerful but allied with the corporations in suppressing workers’ rights. The book manages to explore some complex economics in the context of a well-paced thriller.
Doctorow is clearly fascinated by economic issues, and points out that most science fiction and fantasy economies make no logical sense. The exception, he declares, is when Marxists write science fiction or fantasy. Take the recent Hobbit movie, for example. “How can the goblins have a mine that’s so inefficient?” he laughs, as he pauses from ripping the soft flesh from the marrowbones on his plate with his bare hands.
The porterhouse steak arrives, pre-sliced. It’s very good, charred on the outside but soft and pink beneath the surface. Doctorow has asked for horseradish while I am dipping my steak and chips into béarnaise sauce. The conversation is animated enough to slow our progress, and neither of us raises an eyebrow when a waiter noisily drops something fragile on the other side of the dining room.
So, I ask, if only Marxists get economics right in their novels, does that make Doctorow a Marxist? There’s a tension there, somehow – he’s a successful player in the market economy and fluently speaks the language of business; of profit, marketing reach, margins, and price discrimination. But his political activism seems squarely on the left – pro-labour, pro-equality, pro-rights.
“Marxists and capitalists agree on one thing: they agree that the economy is important. Once we’ve agreed on that we’re arguing over the details,” he says. But no, he’s not a Marxist. “I always missed the explanation of how the state is supposed to wither away.” In his novels and his blogging, the ruthless abuse of state power is just as much of a theme as the grasping amorality of large corporations.
Before long we’re talking about automation, and whether the rise of robots and algorithms is a threat to middle-class jobs. Doctorow’s next book will explore that territory in a suitably dystopian form, and he is keen to pick my brains about how things might play out. We discuss possible scenarios and I recommend an essay by John Maynard Keynes, “Economic Possibilities for our Grandchildren”. (Within hours he’s found it, read it and tweeted a recommendation.)
. . .
We’ve been making sufficiently slow progress through the meal that both of us have room for dessert. In fact, Doctorow effectively orders two – a crumble with cornflake ice-cream on the side. I order peanut butter caramel shortbread. After we both ask for double espresso, he pulls out a small plastic bottle that once contained mineral water. It’s half-full of a pale brown liquid. “I nearly forgot. I brought you some cold brew coffee.” I sniff at the concoction, the product of Doctorow’s latest coffee experiments. It’s made by steeping coffee in cold water overnight, and it smells sweet. When I try it later, the taste is mild but the caffeine jolt is fierce.
As we wait for dessert, I ask him about his recent speeches at technology conferences discussing the “war on general purpose computing”. He runs through the argument with practised fluency. Computers are by nature general-purpose machines. It’s impossible to make a computer that does all the kinds of things we want computers to do yet is somehow disabled from making copies of copyrighted material, or viewing child pornography, or sending instructions to a 3D printer to produce a gun.
“Oh my God, that’s good,” says Doctorow after his first mouthful of crumble. My peanut butter shortbread is fantastic too, if absurdly calorific. We are interrupted only by another waiter dropping a tray of glasses.
He continues with the argument. The impossibility of making limited-purpose computers won’t stop governments or corporations trying to put on the locks, or changing laws to try to make those locks effective. But the only way these limits can possibly work is subterfuge: computers therefore tend to contain concealed software that spies on what their users are trying to do. Such software is inevitably open to abuse and has often been abused in the past.
Digital rights management systems intended to prevent copying have been hijacked by virus-writers. In one notorious case, the Federal Trade Commission acted against seven computer rental companies and the software company that supplied them, alleging that the rental companies could activate hidden software to grab passwords, bank account details and even switch on the webcam to take photos of what the FTC coyly calls “intimate activities at home”. As computers surround us – in our cars, our homes, our pacemakers – Doctorow is determined to make people realise what’s at stake.
We polish off our coffee and desserts, and the conversation rolls on, covering digital media strategy, the future of book publishing, and Rupert Murdoch’s chances of keeping control of News Corp. I ask him about the FT’s business model. He approves of the use of the standard web language HTML5 in the FT app, which makes it less dependent on Apple or any single tablet format. “That’s a good idea,” he says. Then again, he adds, “selling a product that is well-liked by people who are price-insensitive is never a bad thing.”
We’ve been in the restaurant for three hours and are the only customers left. The staff wipe the tables around us and patiently bring flasks of tap water without being prompted. Yet another glass breaks, somewhere on the edge of my vision. “It’s not a good day for gravity,” says Doctorow.
I feel embarrassed that I’m the one who has to call things to a halt, but I’m going to be late for my next appointment. We admire the size of the bill, shake hands, and Doctorow heads off to the pool for a long swim.
That evening, I send him an email. His response is immediate.

157 Commercial Street
London E1 6BJ
Bone marrow & onions £7.00
Doddington Caesar £7.50
Porterhouse 900g £76.50
Triple-cooked chips £4.00
Creamed spinach £4.50
Béarnaise £3.00
Apple crumble £6.75
Peanut butter shortbread £7.00
Cornflake ice-cream £3.00
Double espresso x2 £6.00
Sparkling water x2 £7.00
Moulin de Gassac x2 glasses £12.00
Total (incl service) £162.28

First published in the Financial Times, 13 July 2013


The astonishing life of Bill Phillips

This is the latest video from the recording of my my radio series, “Pop Up Economics“. (Alternative link to watch.)


Thomas Schelling, Henry Kissinger, and Dr Strangelove

The full video of the latest episode of Pop Up Economics (free podcast here). Enjoy and please spread the world. (You can also watch here.)

3rd of February, 2013HighlightsRadioVideoComments off

How to support innovations that matter

That was the topic of the first episode of “Pop Up Economics“, and here’s the video!


How to nurture innovations that matter – Tim Harford live at Wired 2012

Here’s my talk at Wired late last year. It was a really fun event. Enjoy!

(Note, incidentally, that Matt Parker has now moved from cycling to rugby.)


5th of January, 2013HighlightsSpeechesComments off

“The Undercover Economist” – a free chapter

The second edition of The Undercover Economist was published last year in the UK, and recently as an eBook in the US.

The biggest change from the first edition was a new chapter about the financial crisis. Lots of people have written to ask whether they can get this chapter without buying the entire book again. That seems only reasonable, and you can now download it here or as a backup, here. Enjoy.

Previous Next


  • 1 Twitter
  • 2 Flickr
  • 3 RSS
  • 4 YouTube
  • 5 Podcasts
  • 6 Facebook


  • Fifty Inventions That Shaped the Modern Economy
  • Messy
  • The Undercover Economist Strikes Back
  • Adapt
  • Dear Undercover Economist
  • The Logic of Life
  • The Undercover Economist

Search by Keyword

Tim’s Tweets

Free Email Updates

Enter your email address to receive notifications of new articles by email (you can unsubscribe at any time).

Join 160,398 other subscribers

Website Malware Scan
Do NOT follow this link or you will be banned from the site!