If something is being advertised as “incredible,” it probably is.

This post was originally titled, “Asleep at the wheel: Junk science gets promoted by Bill Gates, NPR, Ted, University of California, Google, etc.,” but I decided the above quote had better literary value.

We’ve had a few posts now about discredited sleep scientist Matthew Walker; see here and here, and here, with the disappointing but unsurprising followup that his employer, the University of California, just doesn’t seem to care about his research misconduct. According to his website, Walker is employed by Google as well.

Heart attacks

Also disappointingly but unsurprisingly, further exploration of Walker’s work reveals further misrepresentations of research.

Markus Loecher shared this investigation of some attention-grabbing claims made in the celebrated Ted talk, “Sleep is your superpower.”

Here’s Walker:

I could tell you about sleep loss and your cardiovascular system, and that all it takes is one hour. Because there is a global experiment performed on 1.6 billion people across 70 countries twice a year, and it’s called daylight saving time. Now, in the spring, when we lose one hour of sleep, we see a subsequent 24-percent increase in heart attacks that following day. In the autumn, when we gain an hour of sleep, we see a 21-percent reduction in heart attacks. Isn’t that incredible? And you see exactly the same profile for car crashes, road traffic accidents, even suicide rates.

“Isn’t that incredible?”, indeed. This reminds me of the principle that, if something is being advertised as “incredible,” it probably is.

Loecher decided to look into the above claim:

I [Loecher] tend to be sensitive to gross exaggerations disguised as “scientific findings” and upon hearing of such a ridiculously large effect of a one-day-one-hour sleep disturbance, all of my alarm bells went up!

Initially I was super excited about the suggested sample size of 1.6 billion people and wanted to find out how exactly such an incredible data set could possibly have been gathered. Upon my inquiry, Matthew was kind enough to point me to the paper, which was the basis for the rather outrageous claims from above. Luckily, it is an open access article in the openheart Journal from 2014.

Imagine my grave disappointment to find out that the sample was limited to 3 years in the state of Michigan and had just 31 cases per day! On page 4 you find Table 1 which contains the quoted 24% increase and 21% decrease expressed as relative risk (multipliers 1.24 and 0.79, respectively):

Loecher notes the obvious multiple comparisons issues.

But what I want to focus on is the manipulation or incompetence (recall Clarke’s Law).

To start with, here’s the summary from the above-linked article:

Now, we could argue about whether the data really show that daylight savings time “impacts the timing” of acute myocardial infarction—arguably, the data here are consistent with no effect on timing at all! But let’s set that aside and focus on the other point of their summary: daylight savings time “does not influence the overall incidence of this disease.”

This completely contradicts Walker’s theme of sleep deprivation being dangerous. It did not influence the overall incidence of the disease!

Presumably Walker realized this: even if he didn’t read the whole article, he must have read the abstract, at least to pull out those 24% and 21% numbers. (Or maybe Walker’s research assistant did it, but no matter. If Walker gets credit for the book and the Ted talk, he also gets blame for the errors and misrepresentations that he puts out under his name.)

So . . . he read a paper claiming that, at most, daylight time is associated with some time-shifting of heart attacks, and he misrepresents that as being associated with an increase.

Also, he says is “a global experiment performed on 1.6 billion people,” but he’s reporting results on one U.S. state. He must have realized that too, no, that this was not a N = 1.6 billion study???

But wait, there’s more. We switch to daylight time 2am on Sunday. So you might expect the largest effects to occur on Sunday—that day with the sleep deprivation. Or maybe Monday, the first day back at work. All sorts of things are possible. The point is that that, by saying it as “that following day,” Walker is hiding the choice. If it’d been Sunday, it would’ve been “the very day of,” etc. And then when he talks about autumn, he doesn’t say “that following day,” just leaving the (false) impression that it’s the same pattern both seasons.

Tuesday, huh?

I also wonder about Walker’s other claim, that at the switch to daylight or standard time “you see exactly the same profile for car crashes, road traffic accidents, even suicide rates.”

Exactly the same, huh? I’ll believe it when I see the data, and not before.

Testicles

Loecher also looks into Walker’s claim that “Men who sleep five hours a night have significantly smaller testicles than those who sleep seven hours or more.” There seems to be no good evidence for that one either.

Ted and Edge and all the rest

I hate the whole Ted talk, Edge foundation, Great Man model of science. It can destroy people. As I wrote last year:

Don’t ever think you’re too good for your data. . . .

Which reality do we care about? The scientific reality of measurement and data, or the social reality that a Harvard [or University of California] professor can get caught falsifying data and still be presented as an authority on science and philosophy. Ultimately, both realities matter. But let’s not confuse them. Let’s not confuse social power with scientific evidence. Remember Lysenko. . . .

OK, why am I picking on these guys? Marc Hauser and the Edge foundation: are these not the deadest of dead horses? But remember what they say about beating a dead horse. The larger issue—a smug pseudo-humanistic contempt for scientific measurement, along with an attitude that money plus fame = truth—that’s still out there.

An always-relevant quote

From Dan Davies: Good ideas do not need lots of lies told about them in order to gain public acceptance.

A message to Bill Gates, NPR, Ted, University of California, Google, etc.

It’s not your fault that you got scammed. I mean, sure, it’s kind of your fault for not checking, but that’s easy to say after the fact. Anyone can get scammed. I’ve been scammed! My political scientist colleague Don Green got scammed! Harvard got scammed by that disgraced primatologist. Cornell got scammed . . . George Schultz got scammed . . . maybe you’ve heard about that one. People get scammed. Scammers scam people, that’s what they do.

You got scammed. That’s in the past, now.

The question is: Are you gonna continue to let yourself get scammed?

I’ll break that down into 2 questions:

1. Do you now, at last, realize you’ve been scammed? (If not, what would it take? An actual manipulated graph?? No, I guess not; we already have one of those!)

2. If your answer to the first question is Yes, then are you gonna decide that it’s worth your while to continue to get scammed, because the cost in effort and bad publicity is worse than the cost of continuing to promote this stuff?

If the answer to question 1 is No, that’s just sad, that people could see all this evidence and still not get the point.

If the answer to question 1 is Yes and the answer to question 2 is No, that’s even sadder.

A quick google search appears to reveal six separate appearances by Walker on NPR, with the most recent being this June, several months after Guzey’s takedown of Why We Sleep. But I’m guessing that, once NPR had Walker on once or twice, he became a known quantity for them, so they just put him in the Expert category.

Ted? They’ve had iffy talks before, I guess it comes with the territory. They’re probably focusing on scheduling and promoting new talks, not on problems with talks they’ve already posted.

Bill Gates? He endorsed Walker’s book and now he’s moved on. Gates probably doesn’t care about an endorsement that’s sitting in the past.

The University of California? They know about the problems with Walker’s work but they’ve carefully looked away. I think they’re basically Yes on question 1 and No on question 2, except that they’ve tried really hard to avoid answering question 1. At some level, they must know they’ve been scammed, but as long as they avoid looking at Walker’s work carefully (even to the extent of carefully reading a few blog posts), they can maintain a facade of uncertainty.

Google? I have no idea. I don’t know what Walker does for Google. Maybe he’s doing great work for them. Yes, he has a problem with exaggerating research claims in publications and public talks, but maybe he does excellent work when the lights of publicity are not shining. In that case, he’s not scamming Google at all.

Also, Walker could well be scamming himself. I’m not trying to paint him as some cackling villain here. I could well imagine he’s a true believer in the healing power of sleep, and that when he misrepresents the evidence, in his view that’s just because he doesn’t have all the data at hand. Sure, they didn’t really have data on 1.6 billion people, but if they did, it would undoubtedly confirm his views. He has a direct line to the truth, and he’d be remiss if he didn’t shout it from the treetops.

The trouble is, people who think they have a direct line to the truth, often’t don’t. Recall the above quote from Dan Davies. Tony Blair probably thought those WMDs were real too, at some point—or, if they weren’t, they could’ve been, right? And the war was a good idea anyway, right? Etc. To return to sleep studies: the data support the theory, and once you believe the theory, you don’t need the data anymore.

As always, I’d have no problem if this guy were to just straight-up give inspirational talks and write inspirational books. He could just say that he personally believes in the importance of sleep, and his broad reading and experience has lead him to this conclusion. It’s when he misrepresents data, that’s where I have the problem (even though the University of California doesn’t seem to mind).

11 thoughts on “If something is being advertised as “incredible,” it probably is.

  1. About the DST–myocardial infarction, I think Matt Parker covered it quite well in his video: https://youtu.be/XZGs5Im9f8Q

    I’m still pissed off about the whole misconduct, this is like stirring the knife in the wound but very much needed.
    I was halfway the book when I found Guzey’s analysis through this blog (I think he did a great service to science), and it allowed me to reframe it.

  2. Bill Gates:

    Gates has a lot to say lately about public issues. But his endorsement of this tripe should give us pause about Gates’ opinions. Even extraordinary intelligence doesn’t protect people from bullshit.

  3. I had a dear friend (may his name be a blessing) and I noticed a researcher had the same relatively unusual name, so I thought I’d tease him about ‘his’ research. Except a lot of it was stuff like the risk of leukemia in boys who eat hot dogs, which had results like a band of 16-18 hot dogs a week was associated with an increase but under 16 and over 20 were not. I couldnt make fun of that because it seemed to make fun of itself.

    As for billg, my guess is he sees something of value in the work. I have the same issue with Wansink; I dont give a fig about his data mined and manipulated papers, I actually follow a number of the ideas and they work for me. They’re simple things that speak to common sense: use smaller plates, be aware of how glass sizes fool the eye, move treats out of sight, etc. They’re obvious except when I started using small plates, I got resistance because that’s not the norm. So, I’d be willing to bet that billg sees some value in sleep, perhaps because he had to learn the value of sleep, so he’s willing to overlook bad science.

    This speaks to the bifurcation of messaging. That is, you can speak in common sense terms, as in ‘talk to the people’, but there’s an expectation that this now must be buttressed not only by numbers but by symbolic notation (which very few people bother with or can follow). We should all blame PowerPoint for making slides into a show.

    • This highlights a point that bugs me about social science research: the heterogeneity of human experience. Scientists seem to want to be able to control for some basic things like age/sex/race and say that interventions either work or they don’t.

      So sure, some personal cues might help you as an individual achieve your goals but might have the opposite results in another individual who is the same age/sex/race as you. That doesn’t mean you’re wrong, but it makes me seriously question the research paradigm. Things like that feel more like self-coaching, or what some nerds call “N=1”

  4. From the DST article: “No adjustments were made for multiple comparisons, and the analysis is intended to be exploratory in nature. As such, nominally significant results should be interpreted as hypothesis generating, rather than confirmatory evidence.” So the paper isn’t even claiming that there is time-shifting of heart attacks, it’s claiming only that somebody might want to look into it.

    BTW, my hypothesis is that the loss of sleep at the onset of DST is associated with a 4% reduction in all-cause deaths that very day. The DST article did adjust for that, though.

  5. It may be of interest to those following the Walker story that he recently retracted an opinion piece in Neuron. Apparently he duplicated verbatim several paragraphs from that opinion piece and another piece in Lancet. Of course to submit to Neuron as an author he had to affirm that the work was original, which was clearly false.

    Here is the link to “Retraction Notice to: A Societal Sleep Prescription”:
    https://www.sciencedirect.com/science/article/pii/S0896627320305201

  6. Funny that Matthew Walker appears to be in a thousand times better health than the author of this article. Perhaps his claims are accurate then.

    • Gates:

      Your comment seems off for a couple of reasons. First, I don’t know what it means to be “in a thousand times better health,” let alone how you’d figure this out from a photo. Second, it’s absolutely nuts to evaluate health claims based on the appearance of the advice-giver. What next—taking health advice from Brad Pitt and Gwyneth Paltrow??

Leave a Reply to Martha (Smith) Cancel reply

Your email address will not be published. Required fields are marked *