The continuing misrepresentations coming from the University of California sleep researcher and publicized by Ted and NPR

Markus Loecher writes:

Just when I had put the “Matthew Walker fake news” into a comfortable place of oblivion, NPR sends me this suggested story.

How disappointing that NPR’s fact check is no better than other media outlets. Then again, it is a different TED talk.

I [Loecher] am itching to look into the claims that:

– “Restricting your sleep to 4 hours in just one night, we observed a 70% drop of killer cells“ (minute 10:40)

– “Limiting your sleep to 6 hours for one week, they measured the change in their gene activity profile, they observed (i) a sizeable 711 modified/damaged genes and (ii) half of these were increased, half decreased” (minute 12:40)

Is it not incredible that TED talkers do not have to supply a list of references which they cite ? Every school child’s presentation is required to do this nowadays.

Well, sure, standards are much higher in elementary school. Art Buchwald had a column about that once.

Loecher supplied the background last year:

The video of [University of California professor Matthew] Walker’s 19-minute “Sleep is your superpower” talk received more than 1 million views . . .

I [Loecher] applaud both the outstanding delivery and the main message of his captivating presentation . . . However, at about 8:45 into the talk, Matthew walks (no pun intended) on very thin ice:

I could tell you about sleep loss and your cardiovascular system, and that all it takes is one hour. Because there is a global experiment performed on 1.6 billion people across 70 countries twice a year, and it’s called daylight saving time. Now, in the spring, when we lose one hour of sleep, we see a subsequent 24-percent increase in heart attacks that following day. In the autumn, when we gain an hour of sleep, we see a 21-percent reduction in heart attacks. Isn’t that incredible? And you see exactly the same profile for car crashes, road traffic accidents, even suicide rates.

Initially I [Loecher] was super excited about the suggested sample size of 1.6 billion people and wanted to find out how exactly such an incredible data set could possibly have been gathered. Upon my inquiry, Matthew was kind enough to point me to the paper, which was the basis for the rather outrageous claims from above. Luckily, it is an open access article in the openheart Journal from 2014.

Imagine my grave disappointment to find out that the sample was limited to 3 years in the state of Michigan and had just 31 cases per day! On page 4 you find Table 1 which contains the quoted 24% increase and 21% decrease expressed as relative risk (multipliers 1.24 and 0.79, respectively) . . .

More importantly, these changes were not observed on the “following day” after DST but instead on a Monday and Tuesday, respectively. Why did Matthew Walker choose these somewhat random days? My guess would be, because they were the only ones significant at the 5% level, which would be a classic case of “p-value fishing”. . . .

I was unable to find any backing of the statement on “exactly the same profile for car crashes, road traffic accidents, even suicide rates” in the literature.

I’m reminded of the nudgelord who wrote in the New York Times:

Knowing a person’s political leanings should not affect your assessment of how good a doctor she is — or whether she is likely to be a good accountant or a talented architect. But in practice, does it? Recently we conducted an experiment to answer that question. Our study . . . found that knowing about people’s political beliefs did interfere with the ability to assess those people’s expertise in other, unrelated domains.

Actually, their study said nothing at all about doctors, accountants, or architects.

I guess that for some people, the general news media are a place to exaggerate findings and flat-out make things up in order to get more attention. It’s all for a good cause, right?

What really frustrates me is that when people call them on their, ummm, misrepresentations, these people do their best to pretend the criticism never existed, replying only when absolutely necessary and then not admitting what they got wrong. To Walker’s credit, he seems to have stayed on the (relatively) high road, ducking criticism and continuing to push nonsense, but at least not calling us Stasi, terrorist, etc., or attacking us in other ways. I appreciate this bit of civility.

Loecher wrote the following letter to Ted:

Dear TED organisers,

I am a huge fan of TED talks and find that the vast majority delivers high quality, relevant and truthful content.

At the same time I am quite worried that there seems to be no “fact checking” at all. Given its global reach, its enormous impact and power to influence large number of people, I feel that TED bears a large social responsibility to prevent the dissemination of “pseudoscience”, which you acknowledge yourself:

Is TED full of pseudoscience?

As the global TEDx movement grows, some local events have been targeted by speakers who make unsupported claims about science and health — from perpetual motion to psychic healing. TEDx’s science guidelines clearly state that science and health information shared from the stage must be supported by peer-reviewed research. If you have concerns about the content of a TEDx talk, please write to [email protected] and let us know.

A very worrisome example is the “Sleep is your superpower” talk by Matthew Walker, which received more than 1 million views in the first 72 hours (as of publication, it’s up to more than 11,442,588 !)

There is substantial evidence that Walker is distorting scientific facts and grossly exaggerates, in fact, “lying” would not be an entirely false accusation. If you want to inform yourself on the scandalous assertions, see here, here, and many more . . .

I understand that you probably cannot fact check every talk nor implement a type of peer review. But that is simply not good enough of an attempt to limit this lack of integrity.
One suggestion: Every school child’s presentation is required to supply a list of references nowadays. I feel that would be a minimum requirement for TED talkers as well ? At least, the enormous work currently required to get to the bottom of false claims made in a TED talk, would be reduced.

I would welcome your feedback.

Thanks,

Markus Loecher

I assume their response will be no better than the empty reply we received from the University of California’s time-serving academic bureaucrats (I guess they couldn’t very well admit that they were paying $218,749.00 a year to someone who was doing research misconduct), but you never know. It doesn’t hurt to try.

P.S. I think McBroom’s Law holds here.

22 thoughts on “The continuing misrepresentations coming from the University of California sleep researcher and publicized by Ted and NPR

  1. Has anyone ever seen a published analysis of the frequency or estimated effects of TED talk exaggerations? It would be fun to see the same media hype pick up that story (though it would probably be in the same show or issue as another PR-written story about the latest impossible-to-reproduce “result”.)

    • Eric:

      I dunno. I’m imagining that people send me the worst. I wouldn’t be surprised if most Ted talks are just fine.

      Also, I don’t mind storytelling and exaggeration, as long as the speakers make it clear that’s what they’re doing. My problem is not when they say, “I believe X”; it’s when they say “Science demonstrates X” when it doesn’t. I like Loecher’s suggestion that all scientific or historical claims be given references that are tagged to the appropriate minutes:seconds of the talk. To do that is a bit of work—a Ted employee would have to go through the talk, note all the claims, and then follow up with the speaker to get the references—but Ted clearly has a big budget so I think they could afford this.

      There’s no way that Ted would ever do this, but they should. It wouldn’t solve all the problems—purveyors of bogus unsupported would just supply references that don’t really support what they’re saying, so it would just push the problem back one step—but it would be a start.

      • I recall when you gave a talk at University of Washington Data Science Seminar a few years ago, you said some things about scientists as advocates and that the problem was when they didn’t realize they were advocating. Then more recently I’ve noticed this idea that there will be tendencies to advocate for ideas in science (and that it’s not necessarily bad) in some science reform related philosophy (by Kevin Zollman, David Hull) where the proposal is that science can still make progress even if scientists are sometimes irrationally committed to some ideas.

        Anyway, the idea of scientist as advocate has made me more aware of when my goal is to convey a vision versus when I’m talking about specific evidence, with all of its limitations. I find that often when I give talks, I organize them around whatever my current thinking is for how things in my field need to change, and I focus more on getting across the vision and less on trying to generalize or tell stories about the specific evidence. It’s kind of nice to be able to explicitly say things ‘I’m going to talk about what I think will helpand provide some related evidence but I don’t have all the answers’ so that the audience knows you don’t want what you any of the story to be taken as fact. Maybe there should be more attention to helping early researchers recognize and learn how to openly address this distinction.

      • “I dunno. I’m imagining that people send me the worst. I wouldn’t be surprised if most Ted talks are just fine.”

        On the rare occasions that I get sucked in by the title of a TED talk, it doesn’t take long to see what’s up. I suspect that in general they’re “just fine” in the sense that not much can be directly challenged as blatantly false. They are more like the editorial page of the newspaper, where big chunks of the story are simply left out. When you take a picture on a bright sunny day you just can’t see the stuff in the shade! Not the photographer’s fault!

  2. Language Log used to cite the following Radio Yerevan joke about what it’s often like chasing down the details of these kind of reports (I remember it most in the contact of David Brooks, but TED talks probably are just as apt):

    Question to Radio Yerevan: Is it correct that Grigori Grigorievich Grigoriev won a luxury car at the All-Union Championship in Moscow?

    Answer: In principle, yes. But first of all it was not Grigori Grigorievich Grigoriev, but Vassili Vassilievich Vassiliev; second, it was not at the All-Union Championship in Moscow, but at a Collective Farm Sports Festival in Smolensk; third, it was not a car, but a bicycle; and fourth he didn’t win it, but rather it was stolen from him.

  3. “I was unable to find any backing of the statement on “exactly the same profile for car crashes, road traffic accidents, even suicide rates” in the literature.” Just to a quick google search and you find lots of stuff (not just one off studies).

    Walker certainly manipulates the data he presents, but there are a lot of studies that show increased heart attacks with time change (see https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6463000/ for a meta-analysis). This NYT article has some references for the other stuff (https://www.nytimes.com/roomfordebate/2014/03/06/daylight-saving-time-at-what-cost/daylight-saving-time-takes-a-toll-on-health). Some studies have also been done on the location within time-zone (using variation in sunrise at a given time to identify effects… caveat emptor, obvi). Again, a google search reveals lots of studies from various locales about the effects of springing forward.

    The claim that the spring time change can have some effect on people’s physical and mental health, motor-functioning, and reaction times isn’t that crazy. Presumably it has larger effects on the Monday and Tuesday following because you don’t have to get out of bed and do stuff (like go to work) on the Sunday it occurs, so the effect is delayed ((using variation in church attendance in Sunday church attendance in an area might be one way of testing that). It’s basically a time when everyone gets a case of jet lag. I think it would be odd if we didn’t see something negative occur.

    • Jfa:

      I agree that the general claims (not the effect sizes, but the directions) are plausible, and nothing is stopping Walker from accurately summarizing the literature on the topic. The loss from his perspective would be that he could not make such dramatic claims. For example, instead of claiming a 24% increase in heart attacks in the spring and 21% reduction in the fall, he would have to say a 5% increase in the spring and a 1% increase (not decrease! but, ok, the estimate is noisy so he could say “no effect”) in the fall. That’s a lot less exciting! Indeed, if he were to go back and remove all the exaggerations, it might be that his book would never have been blurbed by NPR, Bill Gates, etc. Maybe no Ted talk either.

      And Walker misreports his own study too. In his Ted talk, he says, “Now, in the spring, when we lose one hour of sleep, we see a subsequent 24-percent increase in heart attacks that following day. In the autumn, when we gain an hour of sleep, we see a 21-percent reduction in heart attacks.” The time change happens 2am Sunday, so “the following day” could be Sunday or Monday; given the results for the spring time change he must be referring to Monday. But then for the fall time change he uses Tuesday! So, first off, the Ted talk is misrepresenting his results in that the sentence is clearly implying it’s the same “following day” in both cases.

      As always, if he’d just be honest with the data, that would be such a step forward. If he wants to give a convoluted reason why it should be Monday and no other day in the spring, and only Tuesday in the fall, he can go for it. He could even do more and come up with a story as to why there’s no evidence of change in any other days. The data he shows are completely consistent with a 5% increase in the spring and no change in the fall, also consistent with no effect at all. That doesn’t mean that I think there’s zero effect, just that these data are too noisy to find the signal he’s looking for. And the claims of 20% effects are just ridiculous. The only way he can get 20% is to scan that table of results and grab the biggest values. That’s how so much of bad science gets done: you generate a bunch of noisy data and then grab the biggest numbers from the pile.

  4. ““Limiting your sleep to 6 hours for one week, they measured the change in their gene activity profile, they observed (i) a sizeable 711 modified/damaged genes and (ii) half of these were increased, half decreased” (minute 12:40)”

    Well, just the wording suggests that the speaker doesn’t know what he’s talking about. You can measure the activity of genes. But gene activity does not provide information about gene damage or modifications. Activity of genes normally varies in response to varying conditions. Gene regulation, the coordinated modulation of activity of different genes at different times and under different circumstances is key to both development and homeostasis.

    You can also detect genetic damage–but it’s a completely different process, and while damaged genes may exhibit different levels of activity from their undamaged versions, changes in activity levels of genes does not imply the genes are damaged.

    • My guess is that he’s only talking about “genes” because people think of genes as some fundamental part of their bodies that should be unrelated to sleep or any other behavior. So he’s invoking “genes” purely for rhetorical purposes, to try to shock people about the importance of sleep.

      The problem is that, as you say, genes turning on and off is an important part of how our bodies normally operate, so it is no surprise that a change in behavior leads to a change in gene activity. Same goes for moving to a new place, exercising, eating something, learning something, etc.

  5. “Indeed, if he were to go back and remove all the exaggerations, it might be that his book would never have been blurbed by NPR, Bill Gates, etc. Maybe no Ted talk either.”

    One of the rules of parsimony is the Law of Least Astonishment: the least astonishing interpretation of the data is the most likely to be true. I would offer that the Ted Talk folks are out there specifically looking for clever findings on interesting topics that violate this law. So yeah, without having sat through more than a couple Ted Talks myself, my prior is that there is a lot of silliness.

  6. What is the least astonishing interpretation of the Bell Correlations at-a-distance that they seem to keep measuring with vanishing margin for error? Super-luminal signaling in space-time? Or instantaneous action-at-a-distance through some extra dimensions which sit above space-time; and when viewed from there, so-to-speak, space-like separated events are actually “next to each other”. (like the surprising large-scale spatial correlations between lighting storms seen by astronauts from above; attributed of course to the large-scale structure of ionospheric currents).

    • I’m not sure why we’re discussing this question here, but I like it a lot. my own personal thought is that the “speed of quantum propagation” could be faster than light. The Bell’s Correlations then would only occur within some radius. However the radius could be quite large. I don’t think anyone has ever done the experiment, but I propose an experiment where we get 3 space vehicles, one in the center, and two “detectors” moving away from the center… We entangle photons, and then send them to the distant detectors. One detector has constant orientation of the detector, the other has orientation that is controlled by a preprogrammed pseudo-random number generator. we make measurements as the detectors move farther and farther apart from the source and see if the Bell inequalities disappear at some radius.

    • Michel:

      No, I don’t think so. I think any population effect would be much smaller. The large effect that was claimed came from one of many comparisons selected from a small sample.

Leave a Reply to Eric Cancel reply

Your email address will not be published. Required fields are marked *