Shira Mitchell, Jeff Sachs, Sonia Sachs, and I write:
The Millennium Villages Project was an integrated rural development program carried out for a decade in 10 clusters of villages in sub-Saharan Africa starting in 2005, and in a few other sites for shorter durations. An evaluation of the 10 main sites compared to retrospectively chosen control sites estimated positive effects on a range of economic, social, and health outcomes (Mitchell et al. 2018). More recently, an outside group performed a prospective controlled (but also nonrandomized) evaluation of one of the shorter-duration sites and reported smaller or null results (Masset et al. 2020). Although these two conclusions seem contradictory, the differences can be explained by the fact that Mitchell et al. studied 10 sites where the project was implemented for 10 years, and Masset et al. studied one site with a program lasting less than 5 years, as well as differences in inference and framing. Insights from both evaluations should be valuable in considering future development efforts of this sort. Both studies are consistent with a larger picture of positive average impacts (compared to untreated villages) across a broad range of outcomes, but with effects varying across sites or requiring an adequate duration for impacts to be manifested.
I like this paper because we put a real effort into understanding why two different attacks on the same problem reached such different conclusions. A challenge here was that one of the approaches being compared was our own! It’s hard to be objective about your own work, but we tried our best to step back and compare the approaches without taking sides.
Some background is here:
From 2015: Evaluating the Millennium Villages Project
From 2018: The Millennium Villages Project: a retrospective, observational, endline evaluation
Full credit to Shira for pushing all this through.
The sad fact is that in the early 2000s many people, including many highly qualified academics, were *begging* Sachs to set up the Millennium Villages in a way that would allow for rigorous evaluation. His response, generally, was “anyone who wants to slow me down hates poor people and wants them to remain poor.” So here we are after spending decades and millions of dollars and the results will never be anything other than inconclusive. But at least Sachs got to hang out with Bono.
Eric:
I have no idea what was happening with the Millennium Villages project in the early 2000s. My involvement began when Shira was hired to do this evaluation project. I’m in no position to evaluate Sachs’s claim that a controlled comparison would not have been feasible. I will say that it’s inevitable for results of such programs to be inconclusive.
Or, to put it another way: I thought the results from our retrospective controlled analysis were pretty conclusive! Some people disagreed with that claim, and they would’ve preferred a prospective controlled comparison. That’s fair enough . . . but I have a feeling that, had we obtained the exact same results with a prospective controlled comparison, the same people would’ve been unconvinced.
The difficulty, as I see it, is that there’s little doubt that the Millennium Villages interventions “work” in the sense of delivering net economic and health benefits to the communities. The debate is whether these interventions are better than various proposed alternatives. And a controlled study would only be able to compare to one alternative, so it would still be fair for a critic to respond that the interventions would not be better than some other alternative that was not considered.
To put it yet another way, just as Sachs was and is committed to the idea that these interventions are a good idea, there are others out there who are committed to the idea that the interventions are a bad idea, and I don’t think a prospective control would convince them. In that sense, the evidence would always be inconclusive.
Still, I agree with you that a prospective controlled design would’ve been better, and we say that in our paper. I think the main advantage of a prospective design is not that it would’ve given conclusive evidence, but rather that the post-study comparison would’ve been a lot easier to do.
Regarding Bono: I have no idea, I’ve never met the guy! It could be that Sachs had fun asking celebrities for money, or it could’ve felt like a chore.
Andrew, I’m sorry for the belated reply. I appreciate your thoughtful response and I hope you’re still monitoring the comments to this post.
It’s important to understand the Millennium Villages in the context of how Sachs was “selling” this idea in the early 2000s. Sachs was advocating something more specific (and, if borne out by evidence, more important) than what your evaluation even considered. Sachs’ theory (detailed at length in his book) was that a combination of simultaneous interventions across sectors (health, education, agriculture, finance…) would put a community (a Millennium Village) on a fundamentally different growth trajectory. Sachs was saying that his prescription of massive, cross-sectoral investments would, in the parlance of the time, release a community from a “poverty trap” of negative feedback, and create a virtuous cycle of self-perpetuating growth. Did your evaluation assess whether that had happened or not in Millennium Villages?
It doesn’t surprise me that your evaluation determined that Millennium Villages “worked” in the sense that if you dump a ton of money into something you’re going to see some movement on the variables you care about. But Sachs was selling something much more revolutionary. If it had worked it would’ve been profound. I don’t think it “worked” on the terms that Sachs sold it, but we’ll never really know because he was too arrogant to even set it up for proper evaluation.
At the end of the day, Sachs’ career over recent decades have proven that someone with impeccable academic credentials can turn out to be little more than an egomaniac, a fraud, and a charlatan. Now he launders propaganda for oppressive regimes in Russia and China. It’s a great relief that nobody in international development takes him seriously anymore, but the cost of his years as an anti-poverty celebrity are immeasurable.
Eric:
Best option would’ve been for Sachs to have talked directly with Putin last year and convinced him that instead of invading Ukraine, to have spent those billions of dollars help African villagers escape a poverty trap. He might’ve even been able to convince the Ukrainians to chip in, considering all the money they’ve saved from not having their country invaded and cities and power stations bombed. Win/win.
Did Sachs get to meet Cher, too? ;)
(Honestly, it was only after reading Andrew’s response that I realized who was meant.)
Gregory:
This is like one of these degrees-of-separation problems:
Connect Cher to Vladimir Putin. It goes Cher – Bono – Sachs – Putin. Just 3 links!
Eric, I’m not familiar with whatever was said at the time, but per the quote above, there were only 10 “main sites.” I can’t imagine you’d have sufficient power with that to do much of an RCT–or maybe there’s some details here I’m missing.
Anyhow, even if you did an RCT here, I’m not sure that you’d have any particular reason to believe these results could generalize to a broader population especially given the multifaceted interventions and all the consequent interactions.
Josh:
Yeah, that was what I was thinking. The prospective controlled trial is better, but not so much because of any randomization, rather because, if done well, it would be associated with better data collection which could be used to do a better analysis.
Unfortunately, I’ve seen examples where researchers used a randomized design to turn off their brains and actually avoid getting good pre-treatment predictors. But that’s the fault of those researchers, not of randomization.
But, yeah, the take-home point is that, yes, a prospective controlled trial would’ve been better, but in a setting like this where the world is changing so much, there will always be plenty of good reasons to declare the results of such a study to be inconclusive.
I have a firsthand account of the MVP – used to work in the project. The MVP was a great idea and Sachs ought to be recognized for putting the plight of the poorest people in the forefront of the global development discourse. But the project was very poorly implemented, and in no part due to Sachs handling of it. I have a firsthand experience in the MVP because I worked on it as a young researcher who was based in the field.
Jeff Sachs was not just the brains behind the MVP but was very hands-on with the project. He was intimately and directly involved with the implementation of the MVP and was informed almost daily on activities in the field. He visited the sites frequently. He led weekly meetings with NY-based and field-based staff, and together with Pedro Sanchez and other senior officials, he directly oversaw the hiring of almost all senior staff in the field.
The problem goes deeper than “prospective controlled design” versus what Andrew and other did. During implementation, it was well known that interventions were not achieving on the ground what Sachs was claiming publicly. You have to recall what was saying in the early aughts, which he articulated well in his book, “The End of Poverty”. During the implementation of MVP, the project was expanding and a positive narrative was needed to ensure that fundraising for the Millennium Promise (MP) and the MVP progressed. (MP was the main fundraising arm of the MP.
There were 2 the main reasons why there was reluctance by MVP management to put in place rigorous evaluation framework. The first is that Sachs genuinely believed that there was no doubt these projects would work as he envisioned them. In a way, Sachs acted as if he was running a clinical trial for a new medicine whereby researchers quickly discovered its high efficacy and thought it was unethical to continue with the study and were justified to simply start dispensing the treatment. The second reason was that there was fear that a rigorous evaluation that published unfavorable results could threaten fundraising. Hardly anyone doubted that money thrown at a problem could solve problems if implemented well. Sachs focused on the money part but seemed obvious to the implementation.
It was simply not true that a controlled comparison could not have been done. Multiple MVP sites were started. The issue of controlled comparison came up shortly after the first MVP site was started in Sauri, Kenya in 2004. There were opportunities to implement controlled comparisons in other sites that started later. I was in MVP at the time. We were clearly aware of this and we all knew that there was no interest by the leadership.
Sachs made it quite clear that he would not entertain any internal discussions on putting in place evaluations. There was no written communication about this – it was verbally and frequently discussed. He was adamant about it and no one in the project could challenge his decision. He was god-like figure in all things MVP. Indeed, everyone in the project owed their positions to him. Sachs was and remains a global figure who was the head of the Earth Institute and had direct access to the UN Secretary General and various heads of state. During Sachs trips to Kenya, Tanzania and Malawi, he was almost treated as a head of state. To give you an idea, when Bollinger once visited an east African country with Sachs during the height of the MVP, many of the local staff had to be told that he was actually Sachs’ boss, not the other way round.
By early 2008, through those weekly meetings, it was clear that many interventions (particularly in agriculture) were failing badly in the sense that were not turning out as planned. These problems were quite obvious in the MVP sites in Kenya and Tanzania. In fact, implementation was so bad that the data obtained from the first household surveys conducted in the first Millennium Village (Sauri, Kenya) to help with monitoring were completely unusable.
I have been aware of at least two internal reports that pointed out that specific interventions failed but were “suppressed”. No suppressed in the sense that the authors were told that to spike the results but rather Sachs and other senior officials gave unmistakable signs it was not helpful to publicly release such results. The staff who worked on those studies were young researchers who were not in any position to challenge Sachs publicly or privately.
When World Bank researchers (see work by Demombynes and Clemens) started to look critically at the MVP results, the response initially by MVP management was simply to ignore them, rather than to engage them. It was only when it became obvious that it wasn’t tenable to ignore them that a new strategy was developed. Unfortunately, MVP’s approach wasn’t to challenge those reports in a way befitting a project run by a premier global institution but rather to simply produce a positive narrative by almost any means. This was necessary to keep up the positive press about MVP so as not to threaten future fundraising. You have to recall that the MVP had no secure long-term funding. Fundraising (led by Sachs) was a constant affair.
This was when Jeff Sachs and other top leaders in MVP tasked Paul Pronyk to do study to get positive results about the MVP. Internally, there was no pretense about doing a rigorous, objective results about the MVP. It was simply to get positive results on MVP. The internal data that showed non-positive results were completely ignored by Pronyk. He duly complied and ended up producing an error-filed report in Lancet that was easily taken apart by others.
The real problem (and the big scandal) in MVP was the poor implementation and poor personnel decisions by Sachs (of which there were too many to list here). These are not problems that are visible to researchers from the outside. Jeff and Sonia ran the MVP as their personal project. For example, the leadership position of the MVP site in Pemba, Tanzania was given to a lady who was a neighbor of the Sachs in New York who had no experience whatsoever in project management or development work. The only background information the East African team knew of the lady was that she was a friend of the Sachs and ran a yoga studio or something similar in New York. The lady was so clueless that it was a long-running joke at the Kenya office that oversaw the sites Tanzania, Malawi and Rwanda. Sachs hired many wrong people for key positions who simply did not perform well. There wasn’t even a background check for some key staff. A key example was Belay Begashaw, who claimed to be a former Minister of Agriculture of Ethiopia, which turned out to be false. Mr Begshaw most of the MVP sites that were located in eastern and southern Africa and was repeatedly promoted by Sachs. A simple google will give you some idea of the man. (https://www.vice.com/en/article/4ad4q3/workers-say-patrons-ignored-non-profit-leaders-culture-of-fear-for-years).
I do not doubt that the MVP led to positive development effects on the targeted villages. After all, millions of dollars were used to provide health, education and agricultural services that were previously unavailable for those poor villages across many sites in Africa. But what Jeff Sachs claimed publicly was quite different from what actually happened in the sites. Let’s take the case of the agricultural interventions. Sachs claimed repeated in numerous settings that agricultural productivity could be achieved through agricultural input credit schemes where farmers’ access to seeds and fertilizers would be pre-financed and they would repay after harvests. Well, the seeds and fertilizers were indeed distributed in Sauri (Kenya) and Mbola (Tanzania) but the repayment rates were atrocious. The highest repayment rates never exceeded 50% in Kenya and were less than 10% in Tanzania. The farmers did report high agricultural yields than before. But the project was never sold (either to the public or donors) as a grant but rather as a viable and sustainable agricultural input financing program. So any retrospective study that look at farm outcomes would have easily found that those farmers benefited but that was not what Sachs claimed publicly and to donors.
In retrospect, Sachs should have simply focused on fundraising and his academic position. He was and remains very good in those things. But he should left the implementation with someone else.
Thanks for taking the time to share your thoughts.
Indeed, our estimation of the MVP’s overall causal effect does not disentangle the effect of the MVP *idea* from the effect of its specific *implementation*. What we’d really want is to compare the effects of various ideas and implementations.
For example, you say the agricultural interventions had a positive effect, but less than promised. I just returned from a visit to the Nelson Mandela African Institution of Science and Technology (NM-AIST), where researchers are comparing agricultural interventions and conditions. We were carefully thinking through experimental designs, trying to control for as many factors as possible. Adding in the complexity such as farmers’ finances would be an important challenge. I’m curious to ask my colleagues at NM-AIST.
No one study gives the full picture, but I hope that clearly-described studies can together form an evidence base to inform policies that improve lives.
That is a very helpful perspective, thank you. You’ve touched on several things such as the pressure on Paul Pronyk to produce a favorable evaluation, whatever contortions that took (which I had assumed to be the case, but had not seen stated), and the farm loan repayment rates (which I’d wondered about, but had not seen figures.)
I imagine you gave some thought to whether you can afford to sign your name to this, and given Sachs’s reputation, perhaps you decided this would create some considerable obstacles in your career, which seems to have many more years to go. I hope you will think about it further.
*Your statements will be much stronger,and will carry more credibility, if they are signed.
*Sachs has clearly decided to divorce himself from discussions of MVP, and seems unlikely to publicly attack critics of his role there, although of course there’s no telling what could happen behind the scenes.
*A great many people recognize that the MVP and Sachs abused their privilege. Might not there be some smaller organizations that would welcome someone willing to speak out?
*A great deal of aid projects are scams, shams, or border on being so. They continue, and they do harm, because those who have the inside story find it easier and more comfortable to simply walk away. Do you feel you have some obligation to help put a stop to this, by going more public with what you’ve seen?
Thank you to all of the earlier contributors. An update on the MVP villages was overdue and this dialogue gives some valuable insights
. To me the crunch issue is not simply are the MVPs better off than they were before. The programme was sold as lifting these villages out of the poverty trap. It was a “proof of concept” and once proven the outlook was that it could be implemented throughout Africa. There is a line at the end of the Sachs/Angina Jolie video about Sauri which says something like “1 village down, 999,999 to go.” The “hook” for donors was that it was a one-off donation. Once they were clear of the poverty trap, the MVPs would have no more need of donations. That made it extremely attractive to institutions that had been giving for 30+ years.
So, my evaluation question is very simple. “How many villages have followed the track plotted by the MVPs? As far as I can find out, the answer is zero. If this is the outcome of a proof of concept, I think anyone can draw the conclusion. Enough said!