Skip to content
 

More bad news: The (mis)reporting of statistical results in psychology journals

Another entry in the growing literature on systematic flaws in the scientific research literature.

This time the bad tidings come from Marjan Bakker and Jelte Wicherts, who write:

Around 18% of statistical results in the psychological literature are incorrectly reported. Inconsistencies were more common in low-impact journals than in high-impact journals. Moreover, around 15% of the articles contained at least one statistical conclusion that proved, upon recalculation, to be incorrect; that is, recalculation rendered the previously significant result insignificant, or vice versa. These errors were often in line with researchers’ expectations.

Their research also had a qualitative component:

To obtain a better understanding of the origins of the errors made in the reporting of statistics, we contacted the authors of the articles with errors in the second study and asked them to send us the raw data. Regrettably, only 24% of the authors shared their data, despite our request being quite specific and our assurances that the authors would remain anonymous. . . .

The paper by Bakker and Wicherts features a truly ugly graph (Figure 2) and also breaks a rule by reporting percentages to inappropriate precision (no, you don’t have to categorize 33/113 as “29.2%”), but I’ll forgive them because I like this sort of work. It’s important and represents a lot of effort. Personally, I think Jelte Wicherts, E. J. Wagenmakers, and John Ioannidis are much more deserving of the ASA Founders Award than is, say, I dunno, Ed Wegman?

15 Comments

  1. John Martin says:

    This brings to mind Feynman’s cargo cult speech. I wonder if statistical errors/ misrepresentations like this increase as you go down the ordering of the sciences from math -> physics -> chemistry -> biology -> psychology -> social sciences.

    • So where does ecology fall on this continuum? Also, since the theoreticians in physics have been so far ahead of the experimentalists for so long, I imagine that would shift their position to somewhere around the social sciences. Does the graph become a cycle?

      • John Martin says:

        Biochemistry -> biology -> ecology -> neurobiology -> psychology, but that’s too simple and linear, right? Most graphs show disciplines as complex, interconnected webs with cycles that change over time.

  2. brenton says:

    @John Martin: what do you mean, “the” ordering of the sciences? Along what metric are you ordering them?

  3. John Mashey says:

    “Personally, I think Jelte Wicherts, E. J. Wagenmakers, and John Ioannidis are much more deserving of the ASA Founders Award than is, say, I dunno, Ed Wegman?”

    Since I don’t know the former (although I’ve heard of Ioannidis and read one of his papers), I think the comment on Wegman is actually not quite fair.

    AMSTAT says:

    “2002: Edward J. Wegman, George Mason University, for over thirty years of exceptional service and leadership to the American Statistical Association; for editing various ASA journals; for selfless and dedicated service to the Statistical Graphics Section of the ASA; for fostering the reputation of the ASA and American statisticians in computing as President of the International Association for Statistical Computing; for significantly expanding the funding for research in probability and statistics at the Office of Naval Research; and for serving in leadership roles at the Interface Foundation of North America since its inception, thus promoting the connection between statistics and computing.”

    I have communicated with some very good statisticians who have expressed:
    a)strong respect for Wegman’s work in the 1970s/1980s, and
    b) serious concerns over behavior starting in early 2000s. As I noted in SSWR p.76, fn 34 about Interface conferences:

    “These conferences seem worthy and interesting interdisciplinary efforts. I might have attended during the 1990s when working at Silicon Graphics, a 1997 sponsor.”

    [As an SGI Chief Scientist, I often tried to find new markets, and had instigated the Operations Research/optimization business at SGI. I made many visits to the Wall Street “rocket scientists” starting ~1995. That led to efforts like Cornell+SGI and So, had I known about Interface, I would likely have tried to attend to get ideas.]

    So, while it is hardly for me to say, the ASA award seems plausibly well-deserved, making the later story a bit tragic.

  4. John Martin says:

    Maybe “down” has a negative connotation – no diss to psychology or social science intended. “Along” or “through” would’ve been better. “Up” is wrong too. Oh the limits of language. Hard/soft, dead/cutting edge, smart/dumb science arguments seem pointless in such a complex world. Your blog is psychology as The Hardest Science, but I can’t tell if you’re serious or it’s tongue in cheek.

    What I want to know is if a discipline’s relation to math is correlated with stats misrepresentation in journals.

  5. […] The (mis)reporting of statistical results in psychology journals – andrewgelman.com […]

  6. […] More bad news: The (mis)reporting of statistical results in psychology journals « Statistical Model… […]

  7. Unfortunately, the situation isn’t any better in medicine. A decade ago my co-authors and I surveyed problems with the use of logistic regression. Having just gotten accepted a letter to the editor of another journal for pointing out exactly the same problems in a recent article, I’d say the medical community still has some work to do in upholding reasonable standards for using statistical methods.

    Bagley SC, White H, Golomb BA. Logistic regression in the medical literature: standards for use and reporting, with particular attention to one medical domain. J Clin Epidemiol. 2001 Oct;54(10):979-85.

  8. Regarding the math -> physics -> chemistry -> biology -> psychology -> social sciences ordering; I think that statistical errors are worse in biology than psychology journals. Psychology majors typically have more exposure to statistics as undergraduates and graduates than biologists (at least laboratory-based biologists).

    Here is another paper looking at statistical errors in biology journals: http://www.biomedcentral.com/1471-2202/11/5

  9. […] I spent four hours with the Monty Hall problem the first time I saw it. I finally realized you should always switch, but I was still uncomfortable with the answer. Others seem to find the answer quite easily. Likewise, there are mistakes people make with statistics that I seem fairly good at pointing out, while others struggle. I have a high aptitude for math, so my inclination is to believe that different types of problems engage different emotional centers of the brain in different people. Not sure. It would be interesting to see a psychological study of some of these problems framed in various ways for different audiences. I probably shouldn't hold my breath, though. About 20% of psychology studies that have been examined by mathematicians show serious errors in, yo…. […]