Skip to content

What’s Wrong with “Evidence-Based Medicine” and How Can We Do Better? (My talk at the University of Michigan Friday 2pm)

Tomorrow (Fri 9 Feb) 2pm at the NCRC Research Auditorium (Building 10) at the University of Michigan:

What’s Wrong with “Evidence-Based Medicine” and How Can We Do Better?

Andrew Gelman, Department of Statistics and Department of Political Science, Columbia University

“Evidence-based medicine” sounds like a good idea, but it can run into problems when the evidence is shaky. We discuss several different reasons that even clean randomized clinical trials can fail to replicate, and we discuss directions for improvement in design and data collection, statistical analysis, and the practices of the scientific community. See this paper: and this one:


  1. Keith O'Rourke says:

    Presume earlier studies are heavily down weighted given absence of blinding – but some brief discussion of this might help lessen “more than a single study” phobia.

    From Frank’s blog post you reference:

    A win: “Interestingly, when the concept [blinded RCT] was explained to patients, they agreed to participate more easily than we thought, and dropped out less frequently than we feared. This means we should indeed acquire placebo-controlled data on interventional procedures.”

    Insurmountable opportunity: “two challenges: (1) there is a lot of between-statistician variation that statisticians need to address, and (2) there are many fundamental statistical concepts that are not known to many statisticians (witness the widespread use of change scores and dichotomization of variables even when senior statisticians are among a paper’s authors)”

  2. george says:

    Re (2), can the presence of change scores or dichotomization with a senior statistician not be explained by them choosing to focus on issues they felt were more important?

    Even senior people have finite resources of time, patience and energy. They may not choose to use them putting the perfect in the way of the possible.

    • Curious says:

      In my experience, a better description is that of putting reality in the way of the pretense. It is far more common to see people pretend that poorly designed studies with small effects are real than it is to accept the reality that the expensive and time consuming intervention had no effect.

    • Keith O'Rourke says:


      I do think its unlikely that a senior statistician would choose to be misled by the familiar phenomenon of “regression to the mean” — if they were aware of it.

      Now I got caught by that same oversight early in my career and when I ran it by the most seasoned applied statistician at the University of Toronto at the time they also missed it. Now, when I debriefed them when the oversight was pointed out to me (by a clinician!), they were very embarrassed.

      Why are these oversights so common as Frank suggest – a curriculum that is mainly about learning math in the context with little to no application context, almost complete lack of professional training, mentor-ship, peer review of on going day to day work, etc., etc. (Statistical review in a non-stats journal is usually a joke.) Until quite recently, most stats depts assigned the weakest faculty member to run the “lab course” and direct the statistical consulting service (3 out of the 4 in my convenience sample).

      I was fortunate to be able to run my work by that seasoned applied statistician for the first dozen years of my work. I also sat through a statistics course almost every term.

      I have met statisticians who have for many years worked with only non-statisticians who believed they knew it all and then changed jobs where other statistician got to see what they did and they have to go through the five stages of denial, anger, bargaining, depression and acceptance regarding their past work and current abilities.

      These days, one can regularly read a blog like this (even some comments) and do much better – well here if they read Andrew’s second linked paper.

  3. Jeff Walker says:

    The stent paper is an excellent case study of much of the discussion on this blog over the years – something useful to hand to colleagues. Can we access Box 2 from the stent paper somewhere?

Leave a Reply