Such questions are among those that fascinate Diego Gambetta. Professor Gambetta, an Italian sociologist based at Oxford University, has managed to wrap himself in the language of economics as capably as Pistone wrapped himself in the language of organised crime. Gambetta is an authority on the Sicilian mafia, but deploys the tools of an economist to understand them and other criminals.

A key concept in modern economics is the “signal”, an idea developed by the Nobel laureate Michael Spence. A signal is an action that distinguishes one type of person from a would-be mimic because it would be too costly for the mimic to carry out. Spence suggested that the decision to acquire a degree might be a signal. The degree may be of no practical value but employers may still value it and quite rationally pay higher salaries to graduates. Why? Because a degree will distinguish good applicants from bad – if bright, energetic candidates are willing to go to the trouble of acquiring one, while dim, lazy candidates are not. The degree serves no educational purpose but the employer uses it to separate the wheat from the chaff.

For a criminal, the stakes are higher and the dividing lines sharper. However similar the boiled-down textbook model might seem, employing a graduate who turns out to disappoint is not the same as plotting an offence with a colleague who turns out to be an undercover cop. But while it is no easy matter to study criminal signals, the danger and purity of the signalling problem that criminals face makes them a tempting group for Gambetta to study in his new book, Codes of the Underworld.

Some signals can be crude. Drug dealers in New York sometimes force customers to use heroin or cocaine at gunpoint. Paradoxically, this has resulted in the signal being weakened: even police can legally take heroin if their lives are in danger.

Other signals seem perverse. Gambetta describes a convention in Italian academia for some established professors to celebrate the poor quality of their published work. This, he says, is a credible signal that they cannot jump ship for somewhere like Harvard – so will remain in Italian academia as powerful patrons.

Joe Pistone managed to avoid such expensive signals and send many smaller and more subtle ones. He didn’t buy drugs, for example, but he did sell stolen jewellery. He turned down a plum job in criminal accountancy, partly for personal reasons, and thus acquired credibility: everyone assumed that no undercover cop would turn down such a job. Above all, he spent years just hanging around in the right company, acquiring credibility by association. It was the long accumulation of many small signals that convinced criminals that he could be trusted.

An interesting implication of Michael Spence’s model is that if degrees really were largely signals, the world would be a better place if universities were closed down. There is an interesting parallel in Gambetta’s work: he points out that prison time provides a wonderfully credible signal. Few undercover police are likely to sign up for four or five years in jail, so an extended prison sentence can be an asset to any criminal trying to establish his credentials. Prisons are sometimes called universities of crime, but surely this is a parallel nobody expected.

Also published at ft.com.

A behavioural economist turned rogue would exploit the “endowment effect” – a tendency for people to put a higher value on something that they feel they already own. He or she would also try to create the sense that consumers would lose out if they did not buy, because people seem to hate the idea of losing £5 much more than they like the idea of gaining £5.

Third, our rogue economist would attempt to suggest an “anchor” value that was much higher than the asking price, which would make the product seem cheap. It doesn’t seem to be hard to create such anchor values: they can be produced by inviting experimental subjects to write down the last two digits of their social security number.

Fourth, he or she would make the pricing as complex as possible so that people struggled to compare one offer with a rival offer. Fifth, he or she would try to create a sense of social approval – everyone is buying this. Finally, a rogue economist would throw in something free.

Many unscrupulous salesmen have figured this advice out for themselves already. Think of infomercials. “The TimCo smokemaster doesn’t retail for £200; it doesn’t retail for £100; it doesn’t retail for £50 … ” (anchoring to a price of £200) … “if our lines are busy, please try later” (social approval) … “the smokemaster is not available in regular stores” (loss aversion) … “but wait! When you buy the TimCo smokemaster you get the TimCo soup knife absolutely free” (complex pricing and use of “free”).

The UK’s Office of Fair Trading (OFT) has been turning to behavioural economists for advice on such tactics, and has found that there is no pricing scheme more pernicious than “drip pricing”. Under the scheme, customers agree to pay a price only to discover that there is a charge for delivery; another charge for paying by credit card, and another for insurance. Drip pricing taps into the endowment effect, because customers feel that they have already made the decision to purchase; it creates loss aversion because customers commit time and effort to the search before being hit with extra charges; and it is a form of complex pricing which makes it hard to compare offers.

The OFT research, conducted by consultants and academics at University College London, was based on a laboratory experiment in which students sat at a computer and were presented with hypothetical deals from two fictional retailers. The students were beguiled with various marketing tricks and had to decide from whom to purchase, in what quantity, and after how costly a search. There was no trick quite so guaranteed to confound them as drip pricing, in which they were hit first with an extra charge for handling and then with a charge for shipping. (A two-part drip is modest: according to the OFT, one package holiday provider used four unavoidable “drips”, and two computer retailers tacked on seven optional ones.)

The OFT has been firing warning shots about drip pricing, but it will have its work cut out to regulate it – there is usually some loophole through which price drippers can slip. Buyers should remember that if they walk away when the drips start to fall, they won’t get soaked.

Also published at ft.com.

The other day, as I was running a sink of hot, soapy water in order to clean a coffee-filter cone, I mused on an inconsistency: we celebrate creating jobs in the wider economy, but complain bitterly about creating them around the house.

We can see the obsession with creating jobs everywhere in public discourse. It seems to be easier to sell renewable energy subsidies through the idea that it will create jobs than the suggestion that it might slow climate change. The coalition’s plans to cut public spending appear to me to be more unpopular on the grounds of lost jobs than lost services. International trade – and before it, new technology, which from an economist’s viewpoint looks much the same – is also condemned because it destroys jobs.

There is much that is silly about all this, and we should pay more attention to the kitchen-sink insight that it’s not a great idea to create needless work. Even if I was inclined to hire a cleaner to wash pointlessly dirty dishes, the apparent job-creation is illusory. The money I felt forced to spend on a cleaner I might instead have spent on a night out, employing cooks and waiters. And even if I had saved it, it would have swollen the pool of savings and made it cheaper for someone to borrow money and set up a business.

Economic growth is a continual process of job destruction. Start with agriculture, which destroyed the jobs of hunter-gatherers, and keep going until you get to e-mail, mobile phones and the word processor, which have destroyed the jobs of secretaries. Historically, some of the people whose jobs have vanished find something more useful to do than the grinding task of finding enough calories: teaching, practising medicine or learning engineering.

In principle, increasing labour productivity (aka “destroying jobs”) could lead to us doing less work for the same material gains. This could be pleasant – welcome to the five-hour working week – or horrible, with an employed elite and an unemployed and marginalised majority. In fact, to the bafflement of yesteryear’s futurologists, we do not lead lives of leisure while robots handle every chore. Instead, we have chosen to enjoy the benefits of greater labour productivity as greater wealth. (We do enjoy more free time too: longer holidays, shorter hours and working lives which start later and finish earlier despite a longer overall lifespan. But we take far less leisure time than we might.)

All that said, there are circumstances in which make-work schemes might make sense. One is the situation in which we find ourselves: a weak economic climate in which public sector job cuts could depress the private sector too. The coalition has a decent argument for making cuts: tax rises would also depress the private sector, while continued borrowing is unsustainable. But the idea that the cuts themselves will help create private-sector jobs is nonsense.

And what of areas whose economies have persistently struggled to recover from the death of an industry? A simplistic economic model suggests that wages will fall, private sector companies will rush in, and growth will resume. Reality suggests a grimmer diagnosis, but not one for which either the left or the right has produced a cure. What is needed are jobs that matter. We don’t yet have a reliable recipe for creating them.

Also published at ft.com.

Don’t come to an economist for the answers. When it comes to choice, classical economics leaps to the punch line and works backwards: if both Betty Sue and Sally Ann are willing to marry you and you chose Betty Sue, the economist can only conclude that you preferred Betty Sue. Whether the marriage made you happy, or you were right to choose her, is none of the economist’s business. Game theory, for instance, may tell you how to get what you choose, but not how to choose.

Economists are not surprised to hear the (perhaps apocryphal) story that circulates about the great game theorist Howard Raiffa, who is said to have been trying to choose between a large increase in salary at Columbia University or a high-status move to Harvard. Sharing the dilemma with a friend, Raiffa was told to deploy the traditional tools of economics. He is said to have retorted: “You don’t understand. This is a serious decision.”

I am not in Howard Raiffa’s league, but I feel his pain when Family Harford wrestles with the big choices in life. The true challenge is that these choices force us to imagine different possible futures – one with Betty Sue and one with Sally Ann; one living in Surrey and one in Scotland; the carefree life of the childless couple versus the nurturing joys of parenthood.

It’s quite possible that our image of these possible futures is not very good. As the psychologist Dan Gilbert points out, you might think that winning the Lottery would make you happier than being permanently paralysed from the waist down, but the empirical evidence suggests that this is just a failure of imagination: paraplegics are not, in fact, less happy than people who have won the Lottery.

To me, that fact was at first surprising, then not very surprising at all. And then, after thinking still further, I realised it kicks away the foundation of almost everything we implicitly believe about the world.

Gilbert’s own experimental work suggests that we are extraordinarily good at “synthesising happiness” – in short, convincing ourselves that we like what we chose and dislike what we rejected, no matter how agonising the choice might have been at the time. This is true even of people with severe anterograde amnesia – an inability to form any new memories. They form much stronger likes for objects which they have previously chosen, despite having no conscious recollection that they had ever chosen them.

So how should we make choices? I recently met Sheena Iyengar, another psychologist and author of a new book, The Art of Choosing – aptly titled because despite a range of recent insights from psychologists and behavioural economists, choice is not a science.

Although she did not phrase it in this way, Iyengar’s method for making big choices boils down to improving the quality of our simulation of the future. First, she says, make a note of your gut reaction. (It may change.) Then list all the pros and cons, as an economist might do. Then talk to other people who have made decisions like the one you are considering – a move to the suburbs, or having a baby. Don’t ask them whether they think they made the right choice: as Dan Gilbert will tell you, of course they do. But ask them about how they live, what they do each day, and what are the advantages and the downsides of their choice.

And then? “Then you go back to your gut feeling,” says Iyengar.

Of course you do. This is a serious decision.

Also published at ft.com.

She may have been more inspired, of course, when she interviewed Tim Smit of the Eden Project itself. “Britain isn’t broken, it’s just a bit bruised around the edges,” he told her. Smit reckons that people simply need an excuse to overcome their shyness.

I generally prefer to wallow in my shyness rather than overcome it, but even I thought this was a good idea. What really struck me was the simple practicality of my wife’s approach. She arranged for the road to be closed, posted notes through letter boxes with her phone number, and told everyone to turn up and bring food and drink to share if they wanted to.

Sure enough, on the day, half-a-dozen strangers emerged from their houses, set up the barriers kindly provided by Hackney Council, and started tying up bunting and putting out chairs. A BMW pulled up at the end of the street. The driver opened the windows and cranked up the stereo before going to set up a barbecue. A gazebo was swiftly erected. A pot of chilli emerged from a nearby house, accompanied by an ice bucket full of beers. The party truly kicked off when the local church emptied out on to the street after a two-hour service, and one of the stalwarts fired up another barbecue. Three hours later, I really did know my neighbours. Everyone is firmly agreed that we’ll do it again next year.

Next year, of course, everything may be a bit grander. We might have a little committee and perhaps organise live music or games. But the truth is that our big lunch didn’t take a whole lot of organising. My wife’s insight was that the thing didn’t have to be planned like a military campaign – or worse, a wedding. When you have a party right outside everyone’s front door, the number of seats, bottles and helping hands expands to fit the situation.

We don’t naturally think this way about getting things done. We expect them to be organised from the top down: first, Tim Smit would convene a committee to decide on the optimal location for a series of street parties; then local managers such as my wife would be appointed to estimate attendance, order supplies and supervise the menu. This would be superfluous at best, and perhaps disastrous.

Head office is not always surplus to requirements: I wouldn’t want to wage a war without a logistics team. But some problems can be solved from the bottom up, and much more effectively. This is most obviously because local people have the best knowledge of the local situation. It’s also because a decentralised approach moves forward unevenly, step by step, in a way that is sometimes hard for a more hierarchical planner to accept. The next street over didn’t have a street party, and that might seem unfair. Not to worry: they saw what we were doing and perhaps they’ll have one next year.

Markets provide bottom-up solutions: nobody is in charge of the supply of bread to London, but the bread gets there somehow. But some market enthusiasts assume that a full-blown market with prices and products and limited-liability companies is the only bottom-up approach. It isn’t, as The Big Lunch showed recently across the country, and as my wife demonstrated on a sunny little street in Hackney – a street whose residents are a little easier in each others’ company now.

Also published at ft.com.

This is one more reason to pity economists. Weather forecasters can glance out of the window if they want to find out what the weather is doing right now. Economists must wait: it took the National Bureau of Economic Research almost a year to declare that the recession had begun at the end of 2007. Economists on the Fed’s Open Market Committee or the Bank of England’s Monetary Policy Committee have the unenviable job of setting interest rates for an economy at whose current state they can only guess.

There’s no mystery as to why “nowcasting” is difficult. The US economy is, to resort to jargon, very big and very complicated. The ideal nowcast would slurp up data on payroll, invoices and payments received in real time from every business in the country. But that asks too much of businesses which are focused on the more pressing matter of giving their customers what they want. Many small businesses simply have no idea how they are doing until several weeks after a quarter has finished, when they figure out how much they owe the taxman. If the business owner herself doesn’t know whether February was a good month until the end of May, it is quite a lot to ask the economists at the Treasury or the Federal Reserve to be more prompt.

Economists continue to wrestle with the problem. A few days before the credit crunch began – more rueful irony – Domenico Giannone and Lucrezia Reichlin of the European Central Bank, with David Small of the Federal Reserve, published a research paper explaining how “nowcasts” of economic growth could be continuously updated as economic data on unemployment, inflation, credit availability and much else trickled out of the offices of economic statistics. David Hendry of Nuffield College, Oxford has, with several colleagues, been working on a particular aspect of the nowcasting problem: spotting a “structural break” – or sharp change in the way the economic variables interact – very shortly after it has taken place. That sounds simple enough, but it is common for forecasting models to miss these structural breaks and then misfire for months or years afterwards.

But if combining official data in new ways is one approach, another is to look for unofficial data sources – such as trending topics on Twitter or the terms most searched for on Google – which are updated much more quickly.

Researchers from Google, including its chief economist Hal Varian and the then head of Google.org, Larry Brilliant, have published research using Google Trends to “nowcast” the spread of influenza, tourist visits to Hong Kong, sales of cars and houses, and unemployment benefit claims. Two economists affiliated with the Bank of Italy, Francesco D’Amuri and Juri Marcucci, have also been tapping into Google data and conclude that it dramatically improves the quality of unemployment “nowcasts” and forecasts – particularly in Italy, where official unemployment data takes two months to arrive.

Real-time data from activity on the web is not perfect, but it’s available almost instantly. I confidently forecast that economists will use more of it in the future. Perhaps they already are.

Also published at ft.com.

Many people fear that ethnic diversity can bring out the worst in us. Most famous is the American political scientist Robert Putnam, author of Bowling Alone, who in 2006 published a study of ethnic diversity and community trust in the US. He told the Financial Times that “the effect of diversity is worse than had been imagined. And it’s not just that we don’t trust people who are not like us. In diverse communities, we don’t trust people who do look like us.”

But while many people will agree with Putnam, few of us ask precisely why ethnicity and co-operation might be connected. Macartan Humphreys, an Irish political scientist at Columbia University, offered me a list of possibilities as to why we seem to get things done more easily in homogenous communities.

First, there are explanations based on preferences. Perhaps “coethnics” want to help each other; or simply like the same things; or perhaps, even if they don’t like the same things, they like working together for its own sake. A second group of explanations suggests that “coethnics” work together more effectively because they understand each other better or can ostracise shirkers.

So far, so imponderable. But Humphreys and three co-authors of a new book, Coethnicity, have been conducting an ambitious series of game theory experiments, not in the psychology labs of the Ivy League but in a neighbourhood of Uganda’s capital, Kampala, which is home to members of many different tribes. The games were designed to distinguish between these explanations. Some are staples of the literature in behavioural economics, others are new. (My favourite is “Lockbox”. First, one subject was trained to open a combination lock, which is an unfamiliar technology in Uganda. The game paired a trained subject with an untrained one and gave the pair 10 minutes to get the lock open without the trained player touching it.)

The results are fascinating. There was good evidence that people from the same ethnic group worked together more effectively (although not at Lockbox, alas) but no evidence that their preferences differed from other ethnic groups or that they harboured any distaste for working with them.

The most striking result came from “dictator” games, in which a player decides whether to assist others who have no way of responding with either reward or revenge. And here’s the remarkable thing: in an anonymous dictator game, people seem to treat those of their own tribe in exactly the same way that they treat people of other tribes. It was only when the researchers lifted the veil of anonymity that people’s behaviour changed and they treated coethnics more generously.

Coethnics, it seems, co-operate in part because that is what they have come to expect from each other – just as the British expect each other to drive on the left – and they are aware of the unpleasant consequences if they do otherwise.

What is true in Kampala may not be true in Hackney – or Jerusalem. But the research is encouraging. It suggests that if we struggle to do business with people who look different, that may not be because we dislike them, but because we simply don’t know quite how to begin.

Also published at ft.com.

I think the professor was right to worry about ministerial exposure to authors such as Malcolm Gladwell, Dan Ariely, Richard Thaler and even Tim Harford – but not for quite the right reasons. The problem is not that such authors are inaccurate. I’m not sure that they are. Gladwell has plenty of critics, but I find him a careful reporter. Ariely is a respected academic; Thaler – also a professor – is widely tipped for a Nobel prize. And what can we say about Tim Harford? I am told he is all but infallible.

Yet infallibility is not enough. It’s perfectly possible for an author to do nothing but weave together credible, peer-reviewed research and yet produce a highly partial view of reality. Different pieces of research invariably point in different directions. Dan Ariely’s Predictably Irrational is full of examples of irrational behaviour. My own Logic of Life is full of examples of rational behaviour. Occasionally I am asked to explain the contradiction, but if there is a contradiction, it is a subtle one.

If Ariely describes a rainy day and I describe a sunny one, we are not really contradicting each other. We each offer our spin, but it’s really about whether most people expect sunshine or rain: Dan says that it’s rainier than we tend to think, while I say the sun shines more often than anyone would credit. A serious review of this metaphorical evidence would count up the rainy days and the sunny ones. It might describe different climates in different parts of the world, the degree of variability, and whether there was any way to forecast rain.

For real policy questions, such reviews exist. They are called systemic reviews. They are increasingly standard practice in medicine, although they are new and scarce in social policy. They should be the first port of call for anyone wanting to understand what works. But they are not exactly bestsellers in airport bookshops.

Quite apart from the fact that nobody wants to read all the evidence, there is a deep problem with the way evidence is selected throughout academia. Even a studiously impartial literature review will be biased towards published results. Many findings are never published because they just aren’t very intriguing. Alas, boring or disappointing evidence is still evidence. It is dangerous to discard it, but let’s not blame Malcolm Gladwell just because he doesn’t stick it on page one.

There’s a hierarchy of evidence here. The systemic review tries to track down unpublished research as well as what makes it into the journals. A less careful review will often be biased towards results that are interesting. A peer-reviewed article presents a single result, while a popular social-science book will highlight a series of results that tell a tale. The final selection mechanism is the reader, who will half-remember some findings and forget the rest.

Those of us who tell ourselves we are curious about the world are actually swimming in “evidence” that has been filtered again and again in favour of interestingness. It’s a heady and perhaps toxic brew, but we shouldn’t blame popularisers alone for our choice to dive in.

Also published at ft.com.

This time the fault was with the paperwork. I showed up, having carefully followed the preparatory instructions I’d been sent, only to be told rather sniffily that I’d followed the wrong ones. “I’ve followed the instructions you sent!” “I didn’t send them.” “Well, I certainly didn’t send them.” Back to square one.

Under the circumstances, I hope you’ll forgive me if I have become strangely obsessed with NHS management.

A fascinating new study by Nick Bloom, Stephan Seiler and John Van Reenen of the London School of Economic’s Centre for Economic Performance (CEP), and Carol Propper of the University of Bristol’s Centre for Market and Public Organisation, attempts to measure the quality of that management. The researchers recruited a team of interviewers who quizzed doctors and managers at 100 hospitals, accounting for 60 per cent of the acute hospitals in England.

They asked open-ended questions about the way the hospital was run. A wonderfully shocking example: “Do staff sometimes end up doing the wrong sort of work for their skills?” “You mean the doctors doing nurses’ jobs, and nurses doing porter jobs? Yeah, all the time. Last week we had to get the healthier patients to push around the beds for the sicker patients.”

The researchers concluded that better hospital management was correlated with better outcomes for patients – modestly so, but in an organisation the size of the NHS hundreds of lives appear to be being lost because of poor management. Bloom and his colleagues also conclude – tentatively, because the comparison is hard – that NHS managers seem to be incompetent compared with managers in the private sector.

It would be nice, then, to figure out what could be done to sharpen up the NHS’s managers and encourage its administrative staff to attach the right instructions to each appointment. One possibility is to bring in lots of those clever private-sector types, but that’s probably a mistake. The evidence suggests that former doctors make good managers, perhaps because they’re better able to communicate with the doctors on their own staff.

A better diagnosis is not that the NHS is missing some elusive quality of private sector-iness, but that it is missing any sort of competitive pressure, the sort of competitive pressure that most businesses (outside banking) have to cope with every day.

The NHS isn’t set up to be a fiercely competitive institution, but the last government did attempt to introduce elements of market reform. Hospitals stand to lose patients or, at the very least staff, to nearby hospitals that are managed more professionally. Isolated hospitals need not worry. As John Hicks once wrote: “The best of all monopoly profits is a quiet life.”

Recognising the iron law of British politics that nobody ever closes down a hospital in a marginal seat, the researchers used constituency boundaries to conduct a kind of natural experiment exploring whether more competition raises management standards.

It seems that it does, a finding that accords with the CEP’s research into private sector management quality, and with new research studying the school system in Florida. Competition raises standards – something that should be music to the new government’s ears.

Also published at ft.com.

The Conservatives point to Sweden and to the US, both of which have introduced policies along these lines. In their manifesto they wrote that in Sweden, “standards have risen across the board as every school does its best to satisfy parents”. The evidence is more ambiguous than that. Early studies of the Swedish reforms looked impressive; more recent work has raised some questions.

In the US, the evidence is more encouraging, but – warns Joshua Angrist of the Massachusetts Institute of Technology – it comes with a health warning. British reformers sometimes talk casually about the benefits of American “charter schools” as though this was some well-defined category. But different US states have very different regimes. Some have simply liberalised entry without much regulatory oversight. Others, such as Massachusetts, are stricter and close down charter schools that are failing to deliver.

The most credible research studies a particular subset of schools, typically in New York and Massachusetts. They are typically “no excuses” schools that emphasise discipline, long hours and short holidays. They are oversubscribed. The disadvantage here is that researchers are looking only at charter schools that parents already reckon are succeeding. But the advantage is that places are allocated by lottery and so researchers can compare lottery winners and losers. The encouraging conclusion is that such charter schools can be dramatically effective, especially for poor children and ethnic minorities.

Several questions remain unanswered, however. One is about how much autonomy charter schools should really have. The ideal combination in the US seems to be freedom from dealing with teacher unions and public school schedules, but nevertheless a tight leash as far as performance is concerned. The schools which give their pupils “no excuses” are allowed few themselves. More laissez-faire approaches to charter schools in other states appear to work less well, which suggests that parent power alone is not enough.

A second question is whether charter schools boost the performance of other schools, hollow them out, or do nothing to them. In Sweden, the signals are mixed. In the US, I am aware of no credible evidence either way. If charter schools are to raise the standards of regular schools, parents need to be able to distinguish good schools from bad. Some new research from Simon Burgess of Bristol University, Ellen Greaves of the Institute for Fiscal Studies, and others, suggests that parents in the UK do sensibly weigh up the academic quality of schools. Parents do like schools with low poverty rates – which might push towards segregation – but they don’t seem to care about race. Poor parents have broadly similar preferences to rich parents.

This suggests that more choice can raise standards in British schools. The Conservative policy is well worth trying. But there is one more obstacle: the Tories need to be willing to shut less successful schools if standards are truly to rise. This has proved a tough sell in Sweden. Will David Cameron’s softer, kinder Tories do better?

Also published at ft.com.