Mike Spagat writes: Here is yet another debunking article I’ve written, this one in the latest issue of Significance. It shows the Lancet once again publishing spectacularly wrong information that has misinformed public discussion on a crucial issue with ongoing … Continue reading
Andrew Mack writes:
There was a brief commentary from the Benetech folk on the Human Security Report Project’s, “The Shrinking Costs of War” report on your blog in January.
But the report has since generated a lot of public controversy. Since the report–like the current discussion in your blog on Mike Spagat’s new paper on Iraq–deals with controversies generated by survey-based excess death estimates, we thought your readers might be interested.
Our responses to the debate were posted on our website last week. “Shrinking Costs” had discussed the dramatic decline in death tolls from wartime violence since the end of World War II –and its causes. We also argued that deaths from war-exacerbated disease and malnutrition had declined. (The exec. summary is here.)
One of the most striking findings was that mortality rates (we used under-five mortality data) decline during most wars. Indeed our latest research indicates that of the total number of years that countries were involved in warfare between 1970 and 2008, the child mortality rate increases in only 5% of them. Les Roberts has strongly challenged these findings.
Michael Spagat notifies me that his article criticizing the 2006 study of
Burnham, Lafta, Doocy and Roberts has just been published. The Burnham et al. paper (also called, to my irritation (see the last item here), “the Lancet survey”) used a cluster sample to estimate the number of deaths in Iraq in the three years following the 2003 invasion. In his newly-published paper, Spagat writes:
[The Spagat article] presents some evidence suggesting ethical violations to the survey’s respondents including endangerment, privacy breaches and violations in obtaining informed consent. Breaches of minimal disclosure standards examined include non-disclosure of the survey’s questionnaire, data-entry form, data matching anonymised interviewer identifications with households and sample design. The paper also presents some evidence relating to data fabrication and falsification, which falls into nine broad categories. This evidence suggests that this survey cannot be considered a reliable or valid contribution towards knowledge about the extent of mortality in Iraq since 2003.
There’s also this killer “editor’s note”:
The authors of the Lancet II Study were given the opportunity to reply to this article. No reply has been forthcoming.
Ouch.
Now on to the background:
More than six-and-a-half years have elapsed since the US-led invasion of Iraq in late March 2003. The human losses suffered by the Iraqi people during this period have been staggering. It is clear that there have been many tens of thousands of violent deaths in Iraq since the invasion. . . . The Iraq Family Health Survey Study Group (2008a), a recent survey published in the New England Journal of Medicine, estimated 151,000 violent deaths of Iraqi civilians and combatants from the beginning of the invasion until the middle of 2006. There have also been large numbers of serious injuries, kidnappings, displacements and other affronts to human security.
Burnham et al. (2006a), a widely cited household cluster survey, estimated that Iraq had suffered approximately 601,000 violent deaths, namely four times as many as the IFHS estimate, during almost precisely the same period as covered by the IFHS study. The L2 data are also discrepant from data provided by a range of other reliable sources, most of which are broadly consistent with one another. Nonetheless, there remains a widespread belief in some public and professional circles that the L2 estimate may be closer to reality than the IFHS estimate.
But Spagat says no; he suggests “the possibility of data fabrication and falsification.” Also some contradictory descriptions of sampling methods, which are interesting enough that I will copy them here (it’s from pages 11-12 of Spagat’s article): Continue reading
I posted a couple days ago on a controversy over methods of counting war deaths. This is not an area I know much about, and fortunately some actual experts (in addition to Mike Spagat, who got the ball rolling) wrote … Continue reading
Mike Spagat writes:
I hope that this new paper [by Michael Spagat, Andrew Mack, Tara Cooper, and Joakim Kreutz] on serious errors in a paper on conflict mortality published in the British Medical Journal will interest you. For one thing I believe that it is highly teachable. Beyond I think that it’s important for the conflict field (if I do say so myself). Another aspect of this is that the BMJ is refusing to recognized that there are any problems with the paper. This seems to be sadly typical behavior of journals when they make mistakes.
Spagat et al’s paper begins:
In a much-cited recent article, Obermeyer, Murray, and Gakidou (2008a) examine estimates of wartime fatalities from injuries for thirteen countries. Their analysis poses a major challenge to the battle-death estimating methodology widely used by conflict researchers, engages with the controversy over whether war deaths have been increasing or decreasing in recent decades, and takes the debate over different approaches to battle-death estimation to a new level. In making their assessments, the authors compare war death reports extracted from World Health Organization (WHO) sibling survey data with the battle-death estimates for the same countries from the International Peace Research Institute, Oslo (PRIO). The analysis that leads to these conclusions is not compelling, however. Thus, while the authors argue that the PRIO estimates are too low by a factor of three, their comparison fails to compare like with like. Their assertion that there is “no evidence” to support the PRIO finding that war deaths have recently declined also fails. They ignore war-trend data for the periods after 1994 and before 1955, base their time trends on extrapolations from a biased convenience sample of only thirteen countries, and rely on an estimated constant that is statistically insignificant.
Here they give more background on the controversy. They make a pretty convincing case that many open questions remain before we can rely on survey-based estimates of war deaths. In particular, they very clearly show that the survey-based estimates provide no evidence at all regarding questions of trends in war deaths–the claims of Obermeyer et al. regarding trends were simply based on a statistical error. The jury is still out, I think, on what numbers should be trusted in any particular case.
Here’s a summary of the data used by Obermeyer et al.:
Michael Spagat has written this paper criticizing the study of Iraq mortality by Burnham, Lafta, Doocy, and Roberts: I [Spagat] consider the second Lancet survey of mortality in Iraq published in 2006. I give evidence of ethical violations against the … Continue reading
