Ethical and data-integrity problems in Iraq mortality study?

Michael Spagat has written this paper criticizing the study of Iraq mortality by Burnham, Lafta, Doocy, and Roberts:

I [Spagat] consider the second Lancet survey of mortality in Iraq published in 2006. I give evidence of ethical violations against the survey’s respondents including endangerment, privacy breaches and shortcomings in obtaining informed consent. Violations to minimal disclosure standards include non-disclosure of the survey’s questionnaire, data-entry form, data matching anonymized interviewer IDs with households and sample design. I present evidence suggesting data fabrication and falsification that falls into nine broad categories: 1) non-disclosure of key information; 2) implausible data on non-response rates and security-related failures to visit selected clusters; 3) evidence suggesting that the survey’s figure for violent deaths was extrapolated from two earlier surveys; 4) presence of a number of known risk factors for interviewer fabrication listed in a joint document of American Association for Public Opinion Research and the American Statistical Association; 5) a claimed field-work regime that seems impossible without field workers crossing ethical boundaries; 6) large discrepancies with other data sources on the total number of violent deaths and their distribution in time and space; 7) two particular clusters that appears to contain fabricated data; 8) irregular patterns suggestive of fabrication in claimed confirmations of violent deaths through death certificates and 9) persistent mishandling of other evidence on mortality in Iraq presented so as to suggest greater support for the survey’s findings from other evidence than is actually the case.

I haven’t read Spagat’s paper and so am offering no evaluation of my own (see here for some comments form a year or so ago), but the discussions of ethics and survey practice are fascinating. Social data always seem much cleaner when you don’t think too hard about how they were collected! May I say it again: a great example for your classes…

P.S. As a minor point, I still am irritated at the habit of referring to a scientific publication by the name of the journal where it was published (“the Lancet study”).

P.P.S. A reporter called me about this stuff a couple months ago, but I’m embarrassed to say that I offered nothing conclusive, beyond the statement that these studies are hard to do, and for some reason it’s often hard to get information from survey organizations about what goes on within primary sampling units. (We had to work hard even to get this information from these simple telephone polls in the U.S.)

2 thoughts on “Ethical and data-integrity problems in Iraq mortality study?

  1. At my institution they tape interviews and code the actual interaction between interviewer and respondent to study the quality of the data we usually take for granted. I like the following (true) example:

    Question: How many times in the last week did you use grain products for breakfast?

    Answer: hmm. never. hmm. no, that's not true; beer is made from grain, isn't it?

  2. Richard Horton, the editor of The Lancet, has been written some strongly worded articles in The Guardian on mortality rates in Iraq. You can read them <a>here (October 2006) and <a>here (March 2007). It is perhaps no surprise that people refer to it as the "Lancet study", when the editor of the journal has been so politically active. Note that, in the first article, he refers to "the Lancet report", which is only one step away from "Lancet study"

Comments are closed.