Is Matthew Walker’s “Why We Sleep” Riddled with Scientific and Factual Errors?

Asher Meir points to this hilarious post by Alexey Guzey entitled, Matthew Walker’s “Why We Sleep” Is Riddled with Scientific and Factual Errors.

Just to start with, the post has a wonderful descriptive title. And the laffs start right away:

Positively Nabokovian, I’d say. I mean it. The above table of contents makes me want to read more.

I’ve not read Walker’s book and I don’t know anything about sleep research, so I won’t try to judge Guzey’s claims. I read through and I found Guzey’s arguments to be persuasive, but, hey, I’m easily persuaded.

I’d be happy to read a followup article by Michael Walker, “Alexey Guzey’s ‘Matthew Walker’s “Why We Sleep” Is Riddled with Scientific and Factual Errors’ Is Riddled with Scientific and Factual Errors.” That (hypothetical) post could completely turn me around! Then, of course, I’d be waiting for Guzey’s reply, “Michael Walker’s ‘Alexey Guzey’s “Matthew Walker’s ‘Why We Sleep’ Is Riddled with Scientific and Factual Errors” Is Riddled with Scientific and Factual Errors’ Is Riddled with Scientific and Factual Errors.” At that point, I’d probably have heard enough to have formed a firm opinion. Right now, the ball is totally in Walker’s court.

After reading various sparkly, hard-hitting bits of Guzey’s post, I was gonna say, “Get this man [Walker] a Ted talk!” But apparently he does have a Ted talk. From Guzey:

I haven’t had this much fun reading something online since the days when Gawker was publishing.

Anyway, it seems that Walker’s Ted talk is called “Sleep is Your Superpower.” Your superpower. How Ted can you get?? The modern world is really wasted on us, without Veronica Geng to mock it for us.

Guzey reports, “Matthew Walker is a professor of neuroscience and psychology at the University of California, Berkeley, where he also leads the Center for Human Sleep Science. . . . His book Why We Sleep . . . was praised by the New York Times . . . was named one of NPR’s favorite books of 2017 . . .”

OK: neuroscience professor at major university, praised by NYT and NPR . . . and he even published an article in the Lancet! We’re touching all the bases here. What next, a collaboration with Dr. Anil Potti?

Guzey continues: “A month after the book’s publication, he became a sleep scientist at Google.”

I’ll have to say, if it’s really true what Guzey says that Walker’s book is riddled with errors, there’s something satisfying about hearing that Walker got a job at Google. We have this image of Google as some sort of juggernaut. It’s good to hear that they can make mistakes too, just like scientific journals, universities, news outlets, etc.

Let me tell you a story. I went to graduate school at Harvard. Finest university in the world. My first day in a Harvard class, I was sitting with rapt attention, learning all sorts of interesting and important things (for reals; it was an amazing class that motivated me to become a statistician), sitting at one of those chairs with a desk attached to it, you know, the kind of chair where the desk part flips up so it’s in front of you, and, on the bottom of that desk was a wad of gum.

Back when I was in junior high, gum was almost a form of currency. I’d buy a pack of grape Bubble Yum for a quarter at the corner store on the way to school, then chew it in the morning during the endless hours between first period and lunch. I’d put one piece of gum in my mouth, chew it until it lost all its flavor, then add the second piece, chew it etc., and continue until I had a massive wad, all five pieces, ultimately flavorless, and I’d chew and chew and blow huge bubbles when the teacher wasn’t looking.

I’m not trying to make myself out into some big rebel here; the point is, we all did that. So of course there was yucky gum under all the desks. You knew to never run your hands under a desk, cos you never knew what might turn up. That was junior high.

Then in high school, everyone was much more mature, a lot less gum chewing . . . but still, gum under the desks. I took classes at the University of Maryland, a fine university with an OK basketball team . . . still, they had gum. Then I went to MIT, one of the finest engineering schools in the world . . . yup, gum. But Harvard? I’d hoped Harvard was better than that. But it wasn’t.

Anyway, that’s how I felt, learning that this purveyor of (possibly) horribly false claims is not just a professor of neuroscience at a top university—we know that top universities have lots of frauds—but was hired by Google. Google! Here I am, almost sixty years old (I don’t feel close to 60, but that’s my problem, not yours), and still there’s room for disillusionment.

Anyway . . . the next question (again conditional on Guzey’s statement that Walker’s book is riddled with errors) is, where does Walker fit in here? Did he not fully read the sources he cited in his book? Did he write much of the book from memory, and just misremember a lot of things? Does he think that it’s ok to get the facts wrong in service of a larger truth? Is he just confused? Maybe he didn’t write the book himself, perhaps he had an army of incompetent research assistants?

Just as an example, here’s the first item in Guzey’s post:

OK, I haven’t followed all the links and read the linked studies. That’s not my job—as Bill James once wrote, I’m not a public utility. If you want to read the linked studies, go for it!

The question is: what was Walker thinking when he wrote things like, “the shorter your sleep, the shorter your life span” or “Routinely sleeping less than six or seven hours a night demolishes your immune system, more than doubling your risk of cancer”? I guess someone will have to ask him. I wonder about this a lot, when people write things that are not supported by the data. It’s my impression that people often write things that sound good, without always thinking about their literal meanings.

Here’s another one:

What was Walker thinking when he wrote, “every species studied to date sleeps”?? Did he forget what he’d read in that 638-page book? Did he read the passage in question so quickly that he took from it the opposite impression? Perhaps he was working from memory, and forgot some things? I have no idea.

OK, I can’t resist. Here’s another:

Again, maybe I’m missing the whole point. Walker should feel free to take a few moments from his jobs at Berkeley and Google to respond in comments to explain how Guzey is misrepresenting the evidence, or if there’s something else going on. I’m open to the possibility.

Or if Walker did get all these things wrong . . . then maybe he can write something to thank Guzey for pointing out the errors. I make mistakes all the time—it happens!—and when I learn about them, I feel bad. Maybe Walker will learn about this and feel bad and figure out how to do better next time. I don’t think Brian Wansink ever got around to thanking people for finding all the errors in his papers, but I think he should’ve.

In the meantime, I appreciate that Guzey wrote his post directly and didn’t bother being polite. Politeness is fine, but it has a cost. First, a polite version of the post would be less fun to read, and less fun means I’d be less likely to read it, which would be a shame, given that this is an important topic. Second, I’m guessing that rewriting this post more politely would take a lot of effort on Guzey’s part, which would mean that maybe he wouldn’t have written it at all, or he would’ve included less information, or that he’d have less time to do other stuff. And that would be unfortunate. Guzey’s time is valuable too.

I wouldn’t’ve wanted Mark Twain, Veronica Geng, or David Sedaris to have worried too much about politeness either. I’m sure Guzey makes mistakes too. That’s ok. Mistakes are mistakes, even when written politely.

And, if it turns out that Guzey got it all wrong regarding Walker’s book, then fine, I’ll report back and update this post accordingly. I wouldn’t be the first person to get fooled by something on the internet.

P.S. Personally, I looove sleep. I sleep 9 or 10 hours a night. OK, I don’t average 9 or 10 hours a night. My average has got to be less than 8, maybe even less than 7, because when I get too little sleep for a night or two, it’s not like I balance it with 11 hours the next night. What I’m saying is that 9 or 10 hours makes me comfortable: it’s how much sleep I’d get if I had no kids and never had an early-morning deadline. Based on my observations of others, I’m guessing I’ll need less sleep as I get older.

P.P.S. Some searching also turned up this amusing exchange:

To be fair, though, my colleagues and I once wrote a book with the subtitle, “Why Americans Vote the Way They Do,” but we never actually answered that question! In the book, all our empirical claims were carefully sourced and explained, but then we just said something silly the title. So don’t trust titles.

P.P.P.S. I had so much fun writing this post, and I’m such a damn perfectionist, I spent 2 hours on it! And now I’m getting to sleep really late, I’ll only get to sleep 7.5 hours at most. So I’ll be uncomfortable and tired in the morning (and I don’t even drink coffee! I never touch the stuff, just don’t like the smell and taste), and it’ll probably lower my productivity tomorrow. Damn. I do feel, though, that if I write what I write when I want to write it, eventually I’ll end up writing the things I should write, the way I want to write them.

49 thoughts on “Is Matthew Walker’s “Why We Sleep” Riddled with Scientific and Factual Errors?

  1. “Second, I’m guessing that rewriting this post more politely would take a lot of effort on Guzey’s part, which would mean that maybe he wouldn’t have written it at all, or he would’ve included less information, or that he’d have less time to do other stuff.”

    Guzey actually addresses that:

    “13. Appendix: “why did you only check Chapter 1?”
    I spent more than 130 hours over the last 2 months researching and writing this essay (~5 hours to write the outline; ~60 hours to get to the first draft; ~65 hours to edit and fact-check), which constituted essentially all of my surplus free time over this time period. Continuing at the same pace, it would take me more than 3,000 hours to check the entire book. 3,000 hours is the equivalent of 75 weeks or 1.4 years of full-time work.

    I hope that going through one full chapter, rather than cherry-picking stuff from across the book, demonstrated the density of errors in the book.”

    The book was apparently so bad that he could only get through the first chapter’s inaccuracies after two months. Imagine if he had to be polite about it!

  2. “I appreciate that Guzey wrote his post directly and didn’t bother being polite.”

    I found Guzey’s article to be quite polite. There were no gratuitous insults or snarkiness, only straight-forward declarative sentences. When he says things like “This is false”, he is simply saying something true in the most straightforward and unadorned way possible. He also includes many caveats which make the text sound very careful, and makes the blunt “This is false” statements that much more powerful.

    I find this style of writing very persuasive. Nothing appears to be hidden or gratuitous. It makes his statements that much harder to counter with obfuscation or misdirection.

  3. My takeaway is another confirmation that very (?) intelligent people, particularly those with significant mathematical bents, tend to stuff gum in their mouths. I have an entire theory based on this, but no TED Talk. The gist is that sticks of gum are groups which are manipulated – masticated, if you prefer – to extract either the maximum density of flavor or the maximum length of flavor. The spatial ability to perceive the taste content of a stick also includes its perceived longevity as a ‘chew’, as a sort of whale skin you work over (do like the Inuit do). And thus the trait is associated not only with spatial manipulations – sticks of gum into mouth, into shapes, etc. – but with attribute manipulations. I think it would be perfect for TED.

    I also note essentially all the respondents – my n is maybe 10 – blow bubbles. This is done whether the gum is typically bubblegum or just gum. As in, you can blow bubbles with Dentene but it takes concentration and the failure rate is high. This is yet another evaluative skill, this time involving analysis of the tensile strength of the gum, something you can generally measure in the gum’s grain.

  4. Long-ish time reader, first time commenter here. Love this, Andrew. Funny story and I can feel your excitement sharing it. Being in the beginning of my career, I especially appreciate the insights on how you, who is someone I really admire, work and think about work.

      • Yeah. When a mathematician says two quantities are equal, they mean they’re exactly equal. When an experimental physicist says two quantities are equal, they mean they’re equal within 10% or so. When a theoretical physicist says two quantities are equal, they’re within about a factor of 10. And when a cosmologist says two quantities are equal, they mean they have the same units.

        • People routinely fail to understand me when I say “these two quantities are basically the same”, you know like $3250.15 and $2920.12 are both basically $3000

          I once had a guy utterly flame me when I said that around 20 standard drinks in a day would kill you… He absolutely insisted that was the most ridiculous thing he’d ever heard, and he’d routinely had more than that in college…

          of course what I meant by that was that the LD50 for vodka shots all at once is about 13, so we can’t say that 20 drinks is in any way “safe”, and since the effectiveness of the liver ranges over about a factor of 3 across the population, and different people have different consumption patterns, even if 20 in a day is near the median in the population, you’d expect to see the occasional person who could drink 40 or so, so as long as DrinksThatWouldKillYouPersonallyIn24Hrs/20 has a distribution with a median around 1 my claim was correct.

          As a philosophy major he didn’t really understand this…

        • I’m not sure if I’m missing the joke here, still kind of groggy and having my first cup of coffee, but I’d say how much alcohol one can* drink during a 24 hour period is drastically different question from how much alcohol one can consume at once. Having been a professional musician I have vast amounts of empirical evidence regarding the former question… “when you’re ridin’ 16 hours and there’s nothing much to do”…

          *I will not be qualifying “can” here.

        • Not a joke, just a fact that with differing consumption patterns, differing liver effectiveness, different health status and soforth, the quantity of alcohol in standard drinks that will kill you is well modeled by a Bayesian distribution like a gamma(4,3/20) which has a peak around 20 and falls off in a tail out to 50 or 60 or so. Even 1 drink might kill you if you’ve taken some medications by surpressing your breathing… so variation is an essential part of this model. you can’t falsify the model by exhibiting an individual who had 28 or 34 drinks in the last 24 hours. for every person like that there’s some person with a defective enzyme or a sensitive brain who wouldn’t even try to have more than single digits

          This idea seems to be particularly odd to the high risk group who are most insensitive to the effects of alcohol. Heavy drinkers tend to stay up late hours and manage their blood alcohol as if they had an IV drip. 12-15 beers a night is routine for this population, but they are quite rare making up 1-2% of the population or so.

        • Ah, gotcha. I’m not against any of that, obviously there’s variation; my point was that the question of “how many drinks will kill you” is next to meaningless without taking time into account. An issue which was touched upon a post or two ago on this blog?

        • Yes, one source of variation is the pattern of consumption. Chugging vodka is FAR more dangerous than having one beer every hour on a schedule. The original discussion was about consumption in 1 day. But as you say things vary in terms of how people would usually do that. There are sort of two main patterns of heavy drinking: binging at a party, and long term management. The long term management people are usually people that do this stuff regularly and have learned how to maximize the time where they are at their preferred fairly high BAC… maybe 0.1% or whatever… these are the most severe alcoholics.

          the binge drinkers at college parties or whatnot would be in much bigger danger consuming the same amount but in a shorter time like 4 or 5 hours. So that all goes into the risk as well.

          This is a topic I find interesting, because it has a LOT of statistical challenges and it has mechanistic modeling components, and it is a serious public health issue, so we have had related discussions here on the blog probably a few times over the last several years. I actually have a bunch of research on this topic from a year back or so that I really should write up… it’s not my main area of expertise though, so in many ways I don’t know how best to go about publishing it.

        • “And when a cosmologist says two quantities are equal, they mean they have the same units”

          Equality of units! Having different units would be a form of inequality!

  5. Re “Every species studied to date sleeps,” I wouldn’t accept that from an Intro Biology student, and I’ve taught it a few times. Most species aren’t vertebrate animals – they’re invertebrates, plants, fungi, bacteria, and so on. Things that anyone interested in sleep studies would have to say “no way you could even define sleep in this critter.”

    Thanks for the entertaining read!

  6. I just googled “wierd (sic) things about how animals sleep” and found this: https://www.countryliving.com/uk/wildlife/countryside/g27810374/animal-sleeping-habits/

    Wow! I hope Walker knows these TED gems about animals!

    According to the link, our nearest relatives, the chimps, sleep on average 9 hours a day. “Paleosleep”! He could have measured their testicles to find out if sleep affects chimp testicles in the same way it affects human testicles. Perhaps, if he dreamed up enough testicle properties to measure, he could have ensured a successfully deployment of Mr. P for a revolutionary insight about testicles and sleep! TED stuff! OTOH, it might be difficult to directly measure these organs, so perhaps it would be easier just to Wansink it. I mean, why leave it to chance to pick which chimp testicle property gets a P-pass, when you can engineer the data and choose the most interesting! I’m envisioning the Wansinkesque experimental apparatus (recorded in the distance in a single grainy photo): a long telescope with giant calipers glued on one end.

    I can already hear Shanker Vedantam announcing the unexpected results on “Hidden Brain”: “testicles aren’t what you think they are. Testicles are really….!!”

  7. Wow, this post and the linked article are excellent. The linked article is the best thing I’ve read this week. I’d really love to see a response from Walker.

  8. So how should Walker respond?

    Walker can simply condescend/sneer/ignore/delay and console himself with his greenbacks (a common and reasonable successful response).

    But is there an honest and reasonably effective response?

    There might be. Walker could start by cataloging all the things that Guzey did not refute. Especially important is to defend the idea that sleep is important. This strategy (tacitly at least) argues that “yeah, sure, maybe there are some weak spots here, but this is important stuff, and don’t throw the baby out with the bathwater.” The objective is to make Guzey sound like a killjoy, opportunistic nitpicker.

    This response might even be an honestly legitimate response. Guzey makes it sound like Walker’s book is overwhelmingly bogus. But is it? Or do Guzey’s criticques apply only to a small fraction of Walker’s points? Guzey’s criticisms are devastating and he there are clearly more than a few serious mistatements. But Guzey doesn’t answer the question of what fraction of Walker’s book is bogus. If Walker is interested in making a serious response, I think that is where he should look.

    • Terry:

      One possibility is that Guzey’s criticisms are way off: for example, if Guzey is actually misquoting from Walker’s book or the other sources. In that case, it shouldn’t be hard for Walker to demonstrate this.

      Another possibility is that Guzey’s criticisms are completely accurate, in which case Walker could thank Guzey profusely for pointing out the errors, broadcast the corrections, and as soon as possible issue a corrected version of his book.

      Regarding your suggestion: Sure, that’s fine, but even if the errors are minor, they should be fixed. If Raghuram Rajan didn’t correct the errors in his book that I noted here (and shared with his publisher before the book came out), I’d be annoyed. One of the two errors I noticed was on the very first page of the book!

  9. And now of course it is in the New York Times, meaning it cannot be dislodged from the upper middle class canon.

    ‘ Matthew Walker, a neuroscientist at the University of California, Berkeley, has this expression, “The shorter you sleep, the shorter your life,” so for the past year and a half I’ve been trying to get better about allocating enough sleep ‘.

    Tech column, Nov. 20, 2019, 9:00 a.m. ET

  10. Walker’s work is now canon, having been referenced in a New York Times tech column. Too late.

    TECH WE’RE USING

    Want the Greenest Device? You May Already Own It

    Nov. 20, 2019, 9:00 a.m. ET

  11. As a blogger, I’ve noticed there is a fairly high positive correlation between the number of hours of sleep I get a (plus amount of coffee drunk) and the number of posts I make the following day: 9 hours of sleep and two large cups of coffee ~ 7 posts. 10 hours of sleep and 3 cups of coffee ~ 9 posts.

    • Scott:

      I agree that by far the majority of the relevant material comes from Guzey’s post, not mine. But:

      1. Not everyone read Guzey’s post; my post brought his post to the attention of more readers.

      2. I read Guzey’s post and endorsed it; that counts for something. To put it another way, when I go to the trouble to research and post on a controversial topic, I appreciate it when outsiders go to the trouble of reading my post and, if appropriate, expressing public agreement with what I wrote.

      3. We learned that there’s a dude who goes around giving Ted talks saying that sleep is your superpower. That’s kinda funny.

      4. As is often the case, we had lots of good stuff in comments. So even setting aside whatever my own content added to the discussion, it gave a space for some good comments.

      5. The gum-under-the-desk story. It’s not much, but it’s something, and it’s the only actual content I added in this post.

      Regarding the waste-of-time issue: yeah, you’ll have to figure out some time-management skills, maybe just skim the first couple of paragraphs of a post before deciding to burn 5 whole minutes of your life reading the whole thing. Indeed, if all this post did was make you realize the necessity of this time-management skill, maybe it was all worth it!

  12. Just stopped the ted talk when I first heard a causal claim about testicles which was obviously not backed up causally (and neither the example he mentioned for sleep deprivation). So I Googled criticism of the books and landed here. Thanks Andrew! Just wanted to leave your first inter-pandemic post!

  13. I just googled “wierd (sic) things about how animals sleep” and found this: https://quotesjin.com/

    Wow! I hope Walker knows these TED gems about animals!
    https://quotesjin.com/nature-quotes/
    According to the link, our nearest relatives, the chimps, sleep on average 9 hours a day. “Paleosleep”! He could have measured their testicles to find out if sleep affects chimp testicles in the same way it affects human testicles. Perhaps, if he dreamed up enough testicle properties to measure, he could have ensured a successfully deployment of Mr. P for a revolutionary insight about testicles and sleep! TED stuff! OTOH, it might be difficult to directly measure these organs, so perhaps it would be easier just to Wansink it. I mean, why leave it to chance to pick which chimp testicle property gets a P-pass, when you can engineer the data and choose the most interesting! I’m envisioning the Wansinkesque experimental apparatus (recorded in the distance in a single grainy photo): a long telescope with giant calipers glued on one end.

    I can already hear Shanker Vedantam announcing the unexpected results on “Hidden Brain”: “testicles aren’t what you think they are. Testicles are really….!!”

  14. He didn’t check his sources—THERE ARE FEW TO NO SOURCES LISTED. Even pop science books have a work cited section, have footnotes. He didn’t even cite his OWN alleged studies. I am still, 4 years later, wondering how this book made it to the printers without any section of works cited or references, and with a major publisher, no less. And wondering how it got to be so successful without someone calling out the fact that it lacks even a semblance of academic rigor. As someone who really does believe inadequate sleep plays an important role in many problems we all face, this book, and the lack of critical response to it, absolutely horrified me.

Leave a Reply

Your email address will not be published. Required fields are marked *