A reduction in error rate of 400-600%: Pretty good, huh?

In comments to the previous post, Alexey Guzey points to this bit from his post on sleep legend Matthew Walker:

In The Lancet, Walker writes:

pilot studies have shown that when you limit trainee doctors to no more than a 16 h shift, with at least an 8 h rest opportunity before the next shift, serious medical errors drop by over 20%. Furthermore, residents make 400–600% fewer diagnostic errors to begin with.

In Neuron, Walker writes:

pilot studies have shown that when you limit trainee doctors to no more than a 16-h shift, with at least an 8-h rest opportunity before the next shift, serious medical errors drop by over 20%. Furthermore, residents make 400%–600% fewer diagnostic errors to begin with (Walker, 2017).

And in Why We Sleep, Walker writes:

several pilot studies in the US have shown that when you limit residents to no more than a sixteen-hour shift, with at least an eight-hour rest opportunity before the next shift, the number of serious medical errors made—defined as causing or having the potential to cause harm to a patient—drops by over 20 percent. Furthermore, residents made 400 to 600 percent fewer diagnostic errors to begin with.

A reduction of 400-600%? 400-600%??? As Guzey notes, you can’t reduce the number (or rate) of errors by more than 100%.

This is Wansink or Wegman-level sloppiness here.

A good reminder, I suppose, that peer review is literally nothing more than review by peers. And of course we can expect your peers to make exactly the same mistakes that you do. They’re your peers, after all! It’s a good thing we have some non-peers around to check things out from time to time.

P.S. Ha ha, you might say, very amusing but this must just be some silly typo or miscalculation; you can’t judge an entire book based on one silly mistake. Then again:

1. It seems that this book had lots and lots of silly mistakes, to the extent that it’s not clear what we’re supposed to believe from it.

2. It’s not clear what the statement was supposed to be. I’m guessing 40-60%: but if that’s really what happened, it would be good to see the data backing it up, and it doesn’t seem that there is any data backing it up.

3. For this same error to appear three different times . . . aren’t the authors reading their own papers? I mean, sure, I too have published embarrassing stuff under my name. But at least I don’t make the identical mistake three times. To paraphrase Oscar Wilde: to make a mistake once may be regarded as a misfortune; to do it thrice looks like carelessness.

19 thoughts on “A reduction in error rate of 400-600%: Pretty good, huh?

  1. I have not checked this example, but when I’ve seen this in my co-workers, it’s something like this, in their explanation:

    The ordinary error rate might be 20%, and it declines to 4%. From 4% to 20% is a 400% increase, so 20% to 4% is flagged as a 400% decrease. It is sometimes hard to convince people they are wrong about this.

    I also see this fallacy in the type of financial advisor you find at branch banks. A 50% decline followed by a 100% increase the next year just leaves you even, not up 25% a year.

    • Agreed, this also struck me as a common error about how to interpret/report percentages.

      This error seems to be intentionally introduced as part of the rhetoric in this case, though. I say this largely because of the surreptitious shift from talking about “serious medical errors” (which are reduced by the more modest-sounding 20%) to “diagnostic errors”, which given that neither of these gets defined suggests that this passage is meant to wow with big numbers rather than actually inform.

      In general, reporting anything but proportions as percentages is a real pet peeve of mine because of the ambiguity you describe in terms of what is getting multiplied (is it percentage of baseline or percent relative to the treatment level?). I’d like to see just the straightforward, “Trainees with mandatory rest periods committed an average of X errors compared to other trainees who committed Y errors” which could then be embellished by, “suggesting that adequate rest can lead to [half/third/whatever] as many errors in medical decision making.” Saying X and Y doesn’t harm the proportional claim, in fact it makes it stronger because we can see where it came from. On the other hand, *not* reporting those numbers clearly makes me suspicious that we are just getting mumbo-jumbo.

      (P.S.: I realize everything I just said was about a single passage from a longer paper that may well define the X’s and Y’s elsewhere, my point is more about how to balance efficiency and honesty in statistical communication in general.)

    • Zbicyclist:

      I was thinking about this, but that doesn’t seem so realistic either, because then they’re saying that the rate declined from 5-7 down to 1 (on some scale). Could diagnostic errors really be declining by a factor of 5 to 7? That seems implausible. And what’s with “400%-600%” anyway? Are there two studies, and one showed a decline by a factor of 5, the other showing a decline by a factor of 7? Hard to believe, especially when no evidence is given. Again, this seems like Wansink territory. (And, again, I’d be interested to read what Walker has to say about this one, if he chooses to reply.)

      • I can totally believe that errors in medical diagnostics would decline by a factor of 5 under some circumstances. you hear about residents who have worked for 24 hours straight with no sleep several times a week for six months or whatever. it’s almost like medical residencies are more of form of hazing than actual training at least in some of these anecdotal stories

        • 24 hours straight with no sleep several times a week for six months or whatever. it’s almost like medical residencies are more of form of hazing

          I have seen/heard second–hand reports of the “I had to do it so these residents can too”.

          It is comforting that commercial truck drivers and aircraft pilots don’t work these hours.

    • zbicyclist noted: “The ordinary error rate might be 20%, and it declines to 4%.”

      In the medical world, there is absolute risk reduction and relative risk reduction. The absolute risk reduction would be

      .2-.04 = .16 = 16%

      But the relative risk reduction would be

      (.2-.04)/ .2 = .8 = 80%,

      which is more impressive and thus, the one usually quoted. Hardly ever quoted is the NNT, the number needed to treat which in this case is (1/.16) a bit over 6.

  2. Yeah that’s basic math with %’s. You can have something increase by an infinite X%, but you can only have things decrease by between 0% and 100%, exclusive.

  3. It isn’t an error unless the intended meaning is incorrect, factually. At worst, it is unclear language. If it is clear to peers, then it is standard language, in that community. The fact that it confuses the naive outsider is a point against the style, but it is unsurprising that such local styles exist.

    • Anthony Kimball:

      A reduction of more than 100% is mathematically impossible for any quantity that is necessarily nonnegative (such as any counted event).

      A fivefold reduction (meaning x became x/5) is an 80% reduction (the change, 4x/5 is 80% of the initial value x).

      There are errors of fact and errors of rhetoric (among many others). Using language sufficiently sloppily that its meaning is not clear is an error of the latter sort, and is evidence of basic intellectual incompetence. It’s generally a bigger problem than errors of fact, because what is easily understood and detected is easily corrected, whereas what is not easily understood and detected is easily let be and is, in practice, difficult to challenge.

  4. One original misreading of 40.0% to 60.0% with the decimal point dropped by someone (a typist?) which is pasted in two other places seems plausible. But that still means poor proofreading (if any at all.) And there’s still the issue of whether there are actually three significant figures in the hypothetical figures.

  5. Shame on you, Andrew Gelman, belittling the accomplishments of these doctors! It’s obvious that what Walker means is that these well-rested residents not only stop making mistakes but also start catching the mistakes of nurses, orderlies, random hospital staff…perhaps even the odd sleep neurologist. You are wrong, and I demand that you correct your post and that you get Guzey to correct his comment. That will be a reduction in errors of 200%!

Leave a Reply to Garrett Cancel reply

Your email address will not be published. Required fields are marked *