The moral case for doing more to help the very poor is unanswerable. The practical case is more problematic: much foreign aid has been spent poorly in the past and we still have plenty to learn.
No one has done more to draw attention to the moral case than Columbia University’s Jeffrey Sachs. But some other development economists argue that Professor Sachs’s flagship project, Millennium Villages, has passed up a chance to advance our knowledge of what works. A new report, written by Michael Clemens of a pro-aid think-tank, the Center for Global Development, and by Gabriel Demombynes of the World Bank’s Kenya office, criticises the Millennium Village programme for a lack of rigorous impact evaluation.
The Millennium Villages are over a dozen clusters sprinkled across Africa, with an average population of about 40,000 people. The basic idea is that an intensive package of aid – fertiliser, agronomical advice, mosquito nets, school meals, clinics, irrigation and more – can transform the lives of these very poor people. The project aims to become self-sustaining over time, so that villagers can “get on the ladder of development and start climbing on their own”. Another important element is the multifaceted intervention: Sachs says this is necessary because there will be “important synergies”.
Given the complexity of the aid package and the small number of pilot projects, it’s not feasible to evaluate every possible element of the villages with the gold standard of a randomised trial. But several evaluation experts told me that it would have been possible to evaluate the impact of the entire project by randomly assigning control and treatment villages, which would have provided a measure of whether the villages were achieving their goals.
Why might this matter? China’s Southwest Project, supported by the World Bank in the late 1990s, also took a multifaceted approach. At the time, the project looked terrific, but a later evaluation by the World Bank showed that non-project villages largely caught up with project villages within five years. The Millennium Villages hope for a longer-lived impact, but Clemens and Demombynes claim that the current evaluation methods in place will not determine whether they succeed.
They point out that the evaluations published so far on the Millennium Villages website offer simple before-after comparisons showing what’s happening on the ground. These are not the only plausible indicators of success. Demombynes says, “It’s very clear that you can’t use a simple before-after comparison, especially in Kenya.” For example, the evaluation shows that mobile phone use in the first village, Sauri in Kenya, has quadrupled. But, says Demombynes, “Mobile phone uptake has quadrupled in Sauri, in the region, and in Kenya as a whole.”
Adjusting for regional trends, Clemens and Demombynes conclude that the impact of the villages is real, but roughly half what the project’s backers claim. And they are worried about sustainability. They stress that their alternative estimate is itself imprecise compared with a true randomised trial.
Jeffrey Sachs robustly defended the Villages when I asked him to reply: he said it wasn’t appropriate to try to control for regional or national trends. Clemens and Demombynes, he wrote in an e-mail, “don’t seem to realize that the interventions we are pursuing in the MVs are also taking place, albeit more sporadically … in the rest of the country.”
Sachs and his colleagues argue that we will learn a lot from the ongoing process of managing, measuring and observing what works within the villages themselves. I agree. But I think it would have been possible and desirable to learn much more.
Also published at ft.com.