Matt Bogard writes:
I don’t have full access to this article to know the full details and can’t seem to access the data link but with n = 49 split into treatment and control groups for these outcomes (also making gender subgroup comparisons) this seems to scream, That which does not kill my statistical significance only makes it stronger.
From the abstract:
Results: Theoretical and practical training in cold immersion in the winter did not induce anxiety. Regular cold exposure led to a significant (p=0.045) increase of 6.2% in self-perceived sexual satisfaction compared with the pre-exposure measurements. Furthermore, considerable increase (6.3% compared with the pre-exposure period) was observed in self-perceived health satisfaction; the change was borderline significant (p=0.052). In men, there was a reduction in waist circumference (1.3%, p=0.029) and abdominal fat (5.5%, p=0.042). Systematic exposure to cold significantly lowered perceived anxiety in the entire test group (p=0.032).
Conclusions: Cold water exposure can be recommended as an addition to routine military training regimens. Regular exposure positively impacts mental status and physical composition, which may contribute to the higher psychological resilience. Additionally, cold exposure as a part of military training is most likely to reduce anxiety among soldiers.
I’m not planning to pay 42 euros to read the whole article (see image above), but, yeah, based on the abstract it looks like any effects here are too variable to be discovered in this way. This one hits a few of our themes:
1. Lots of p-values around 0.05. Greg Francis has written about this.
2. Forking paths: lots and lots of different ways of slicing the data.
3. Small sample size. N = 49 isn’t a lot even before getting into the subgroups and interactions.
4. Implausibly large effect-size estimates. An average reduction of 5.5% of abdominal fat, that sounds like a lot, no? This problem comes for free when variability is high.
5. Noisy measurements that don’t quite align with questions of interest. I can’t be sure about this one, but I’m not quite sure that the life satisfaction and sexual satisfaction surveys are really measuring what’s important here.
6. Story time. Even setting aside the statistical problems, do you notice how they move from “sexual satisfaction,” “health satisfaction,” “waist circumference,” and “abdominal fat” in the Results, to “mental status and physical composition” in the conclusion? I guess “getting skinny and having good sex” wouldn’t sound so good.
7. Between-person comparisons. There’s no need for this study to be done in this way—some people get treatment, some get control. It should be easy enough to do both treatments on each person, but it seems that they didn’t do so. Why? I guess because between-person comparisons are standard practice. They’re easier to analyze and at first glance look cleaner than within-person comparisons. But that apparent cleanliness is an illusion.
8. Coherence with folk theories. Cold showers! Sounds paleo, huh? I’m not saying that cold showers can’t have benefits, just that this is a noisy study with the sort of conclusion that a lot of people will be happy to hear.
What’s going on?
I don’t know what’s going on. Here’s my guess: These researchers took some measurements that vary a lot from person to person, maybe some of these measurements vary a bit within person too. They applied a treatment which will have variable effects: maybe very close to zero in some cases, positive for some people, negative for others. Given this mix, we can expect the average effects to be small. Small average effects, indirect measurements, high variation . . . it’ll be hard to find any signal amid all this noise. Then this gets piped through forking paths and the statistical-significance filter and, boom!, results come out, ready to be published and publicized. I’m not saying the authors of the paper did anything dishonest, but that doesn’t stop them from pulling comparisons out of noise.
It’s the usual story of junk science, the push of thousands of journals seeking publications and millions of people doing research, combined with the pull of “the aching desire for an answer” (as Tukey put it) to unlimited numbers of research questions, mixed in with the horrible ability of statistical methods to convince people there’s strong evidence even when it isn’t there.
The article in question was from an obscure journal and I figured I’d never hear about it again.
Part 2
But then I checked my email, and two days earlier I’d received this message from Scott McCain:
Some friends and family are into the idea of cold showers. I’ve seen some work on it before. Recently, Stanford professor Andrew Huberman has covered this study supporting that cold exposure can have a whole host of benefits (including self-perceived sexual satisfaction and reduced waist circumference). I felt that this was interesting but a bit surprising.
I don’t have access to this study, so I downloaded the raw data—which is great that they published it! It seems like they’ve measured a whole bunch of things. I’ve tried communicating to friends and family that this study (however I haven’t analyzed their data myself, besides a cursory look) seems likely underpowered and maybe has been at risk of a garden of forking paths.
Indeed. Again, I’m not saying that cold or hot showering has no effect; I just don’t think this sort of push-button model of scientific inquiry will be useful in figuring it out. But, just to be clear, I’m not trying to talk your friends and family out of taking cold showers. They should go for it, why not?
Part 3
And then I received another email, this one from Joshua Brooks, pointing to a series of twitter post from Gideon Meyerowitz-Katz slamming the above-discussed study. It seems that the cold-shower paper became widely discussed on the internet after it was promoted by Andrew Huberman, a neurobiology professor at Stanford who has a podcast and a “once-a-month newsletter with science and science-based tools for everyday life.”
We’ll get back to Huberman in a moment, but first let me discuss the posts by Meyerowitz-Katz, who writes that the paper in question “shows precisely the opposite” of what it claims. I wouldn’t put it that way; rather I’d just say the paper provides no strong evidence of anything. It’s a noisy study. Noisy and not statistically significantly different from zero is not the same thing as saying that the effect is not there or even that the effect is not important; it’s just that the study is too weak to find anything useful. Also, Meyerowitz-Katz is annoyed that the paper focuses on before-after comparisons. But before-after comparisons can be fine! You learn a lot by comparing to “before” data. And in any case you can compare the before-after differences in the treatment and control groups. On the other hand, Meyerowitz-Katz comes to the same conclusion that I do, which is that the study appears to be consistent with null effects so its conclusions should not be taken seriously.
OK, one more thing. Meyerowitz-Katz writes:
To sum up – this is a completely worthless study that has no value whatsoever scientifically. It is quite surprising that it got published in its current form, and even more surprising that anyone would try to use it as evidence.
I wouldn’t quite put it that way. First, who’s to say it’s “completely worthless”? It has some measurements and maybe they’ll be useful to someone. They posted their raw data! Second, I’m fine with saying that it’s too bad that the paper got published or that anyone would try to use it as evidence. But to call this quite surprising?? Bad or pointless research papers get published all the time, in all sorts of journals, and then they get taken as evidence by all sorts of people, renowned professors and otherwise. So I’m surprised Meyerowitz-Katz is surprised. His surprisal suggests to me that he puts too much faith in journal articles!
Anyway, after looking this all over, I responded to Brooks:
I guess “BMJ Military Health” is a pretty obscure journal . . . but, sure, lots of bad stuff gets published! I’ve never heard of this Huberman guy. I guess if this thing of hyping crap science works for Gladwell, NPR, and Ted, it makes sense that people with less elevated perches in the media will try it too.
I’m not trying to be cynical here—I don’t think that hyping crap science is a good thing—I’m just trying to be realistic. There are lots of journals out there, and if you fish around through enough of them, you can find superficially-plausible articles that will support just about any position.
Brooks followed up with further background on Huberman:
On the cold showers, he points to a meta-analysis.
I don’t know, actually, that he bases his view to any significant extent on that one paper.
The cold immersion claim is just one of the rather remarkable claims he makes on a whole range of effects…
They mostly come off as credible at first glance to me. For all, he claims an “evidence base” in the literature. Personally, I don’t necessarily dismiss everything he says per se, but when i string together the sheer number of absolutely certain claims he makes about such large effects, I have to conclude there’s a fundamental flaw.
Also, apparently he hawks supplements, plus, I’ve also hear him hawking such things as mattresses customized to fit individual consumers by responses to an online questionnaire, that result in improved sleep.
I’m currently listening to a podcast where he’s talking to a scientist about the genetics of “inherited experience.” Right now they’re describing experiments showning a differential effect to worms who are fed other worms who were exposed to an experimental condition (electric shock) then put into a blender.
It’s actually pretty interesting – and some of the research they’re talking about supposedly has been replicated.
But it all feels kinda like the ESP research.
It’s hard to think about these things because there could be real effects! As discussed above, to the extent that cold showers have meaningful effects on people, we should expect these effects to vary a lot from person to person.
I went to Huberman’s webpage on the cold showers to see the meta-analysis that Brooks mentions, but the only meta-analysis I found there was “Impact of Cold-Water Immersion Compared with Passive Recovery Following a Single Bout of Strenuous Exercise on Athletic Performance in Physically Active Participants: A Systematic Review with Meta-analysis and Meta-regression.” Cold-water immersion for athletic performance seems to have zero overlap with cold showers for mood and general health. Nothing wrong with talking about this study but it doesn’t really seem relevant for the discussion of cold showers.
Also Huberman has this:
Building Resilience & Grit
By forcing yourself to embrace the stress of cold exposure as a meaningful self-directed challenge (i.e., stressor), you exert what is called ‘top-down control’ over deeper brain centers that regulate reflexive states. This top-down control process involves your prefrontal cortex – an area of your brain involved in planning and suppressing impulsivity. That ‘top-down’ control is the basis of what people refer to when they talk about “resilience and grit.” Importantly, it is a skill that carries over to situations outside of the deliberate cold environment, allowing you to cope better and maintain a calm, clear mind when confronted with real-world stressors. In other words, deliberate cold exposure is great training for the mind. [Boldface in the original.]
“Grit,” huh? C’mon dude, get real.
P.S. Here’s the supplement he’s advertising:
Looks a little bit iffy, but, hey, what do I know? I’m never studied human performance. I kinda wonder if Huberman takes these himself. I could imagine a few options:
1. Of course he takes them; he’s a true believer.
2. Of course he doesn’t take them; the sponsorship thing is all about the money.
3. He believes they work, but he doesn’t think he personally needs them, so he doesn’t take them.
4. He doubts they do anything, but he figures they won’t hurt, so why not, and he takes them.
Maybe there’s some other option I haven’t thought of.
P.P.S. The research paper on cold showers has been retracted.
The link to the “raw data” is broken.
Nice touch that this post appears immediately above the prior post on “Cheating in science,….” It almost looks like the posts are connected (which they are).
One of the worst consequences of this kind of cheating in science is that this stuff is virtually free to produce and it drives out grant money to fund expensive real science. Why fund someone to spend 15 years looking at cellular level metabolic control of body fat when a couple cold showers will reduce your body fat by 5.5% ? Heck that would cure the obesity epidemic right away!
I haven’t looked at the material just what’s quoted here but I interpreted that to mean 5.5% body fat reduction, but probably it only means 5.5% reduction in body fat percentage. So if you have say 10% body fat then 5.5% of 10% is 0.55% of your body fat. So yeah maybe cold showers for a couple weeks can drop you from 10% body fat to 9.45%
In any case this kind of bullshit noise mining is an industry, there are thousands and thousands of practitioners, and it’s driving a major shift in investment in real science. Not all of it is this hypey, if you just read grants funded by the NIH you find all sorts of stuff like this, GWAS studies of whatever, and people noise mining the microbiome and all kinds of similar stuff. People doing studies where they carefully control conditions in mouse surgeries and detect interactions between different cell types during the repair of certain injuries can’t get funding because “not innovative enough” or whatever.
Daniel –
> In any case this kind of bullshit noise mining is an industry,…
Huberman’s podcast is extremely popular, which means extremely lucrative, although apparently he pumps a “significant amount” of the proceeds back into his research endeavors.
https://hubermanlab.supercast.com/
I’m both fascinated and queasy about this whole ecosphere, where there are a lot of links back and forth between folks like Lex Friedmann (see next comment below), and Joe Rogan, and Jordan Peterson, and Tim Ferris, and Bret Weinstein…
Part of the gestalt seems to be that they’re all extremely complimentary to one another and, in my eyes, don’t apply much scrutiny.
It feels like an industry to me (one that’s different than the one you’re referring to), that overlaps between science and public health and politics and I feel a combination of being pissed off and jealous that people are making so much money from pushing hot takes on all these subjects of dubious validity (I don’t make piles of money from my hot takes!). But part of my reaction is political or ideological and also maybe I should be less rigid and more open to evidence that breaks into new territory.
The recent brouhaha arising from RFK Jr’s recent appearance on Rogan, (and also with Friedman and Weinstein and Peterson) highlights the whole phenomenon. It’s a very large ecosphere but for many people it’s as if exists in a parallel universe that they never see…
Joshua:
Your phrase, “they’re all extremely complimentary to one another and, in my eyes, don’t apply much scrutiny,” applies to lots of academia too. Not 100%, but a lot, so you get big shots promoting each other, getting defensive about outside criticism, and promoting the next generation of researchers to follow in their footsteps. Remember when James “racist” Watson said that a cancer cure was coming in 2 years? That’s how these guys talk: all their friends are beautiful geniuses, everybody’s a hero, etc.
I think that one reason that those psychology professors got so annoyed at Uri Simonsohn, me, and others is that they (the people who got annoyed at us) were used to academic logrolling and adoring media coverage, so it seemed to them as if we were breaking the rules. Why were we such haters? It’s the Javert paradox.
Rogan, Peterson, etc.: they’re another story. They’re entertainers and promoters, and they need a constant supply of new material. People like Matthew Walker and Andrew Huberman are essential to these podcasters, as they supply essential content for the mill.
Huberman, along with the Freakonomics bros and the Nudgelords, are somewhere in the middle: They produce new material that they’re busy promoting, and they also promote work of others, which they also want to promote. They have little incentive to be skeptical about anything.
P.S. I suspect if you would ask Walker, Wansink, Huberman, etc. what they think of Simonsohn, me, etc., they’d probably have two reactions:
1. Why do these capable researchers (Simonsohn, me, etc.) waste time on criticism rather than on positive research? Life is too short to waste it on criticism!
2. The specific criticism of their (Walker’s, Wansink’s, Huberman’s, etc.) work is a bunch of technicalities. Sure, there might be some valid statistical points, but, ultimately, who cares about statistics? For us (Walker, Wansink, Huberman, etc.), statistics is not an end in itself; it’s just a tool, and our goal is to advance the frontiers of knowledge and to help people, so we don’t want to waste our time in arcane statistical disputes.
My responses to these (imputed) reactions are:
1. We learn a lot from seeing what went wrong with bad research. Studying research pathology can help us improve healthy research. But, yeah, sure, there’s a division of labor, and we (Simonsohn, me, etc.) don’t expect or demand that every scientist or most scientists be critics-at-large. We would, however, like all scientists to be vigorous critics of their own work. And if criticizing others’ work can be a step forward toward that goal, that’s good too.
2. It’s these researchers (Walker, Wansink, Huberman, etc.) who are making statistical claims and trying to take data from their little studies and generalize to future outcomes in the larger population. If they just want to present qualitative findings and then speculate, they should go for it! They invite statistical criticism by making strong claims based on statistical statements.
Yikes –
Very much linked the wrong tweet. This is what I mesnt to link.
https://twitter.com/hubermanlab/status/1677530759777902592?s=20
That reminds me of this one:
https://twitter.com/CassSunstein/status/1478522425663012864
Probably the most fun he had until he went to Kissinger’s 100th birthday party.
Andrew said:
‘Your phrase…applies to lots of academia too.’
Badabing. And activism. And business. And politics. Mutually beneficial bullshit promotion everywhere you look.
> That’s how these guys talk: all their friends are beautiful geniuses, everybody’s a hero, etc.
I work very differently – it’s hard for me to understand and I have a negative reaction. When I hear them constantly praising how wonderful they all are…it strikes me as off and is off-putting. As you describe below, to optimize (Huberman uses that term very frequently) science, accepting flaws in research is critical to improve research.
But I guess I can’t say that there’s some objective reason why they shouldn’t praise each other so much. It certainly pays handsomely and a whole lot of people feel very positively about their product.
Huberman has now invited criticism about his romantic affairs by making strong claims to his overlapping romantic partners. It seems he treats his romantic affairs the same way he treats science. Unfaithfully.
Christopher is apparently referring to this article that came out recently in “New York” magazine. I had a bad feeling about Huberman to begin with. https://nymag.com/intelligencer/article/andrew-huberman-podcast-stanford-joe-rogan.html
It would seem that Stanford University is, similar to Harvard (which is nowhere as good in football), an ever presence in this blog; but with regard to Huberman, here he is with the noted scientist, Joe Rogan:
https://open.spotify.com/episode/0ScxTKuC8EaZpIxAfWZNpu
As far as I can tell, perhaps because he too peddles supplements, Huberman has not yet appeared on Alex Jones’ Infowars.com.
I don’t like cold showers… consequently this can’t be right. There. Meta-analyze that.
I don’t, either. But I do have to cool down in the shower, i.e. gradually lower the temperature until it’s almost tap temperature (in summer). But speaking of cold showers, this stuff is nuts. Well, at least it’s good for rats…
https://theincidentaleconomist.com/wordpress/does-cold-immersion-therapy-work-lets-dive-in/
I listen to Huberman’s podcast every now and then on my commute to work. He does often come across as a faithful follower of the ‘church of statistical significance’, no big surprise that he is a fan of Matthew Walker. In Huberman’s defense, he does often tell listeners that the best thing they can do for their health is to make sure they’re getting enough sleep, and that no supplement is a replacement for that. Personally, I have the same point of view as Andrew, if you’re interested in this stuff you may as well try it and see if it works for you. Who cares if there’s no effect on average when it’s very likely the effects are heterogeneous
“Who cares if there’s no effect on average when it’s very likely the effects are heterogeneous”
Because that would suggest that there are negative effects on about half of people who use them?
I should’ve been more careful with my wording, that’s my mistake. However, even if there are negative effects for half of the people, so what? A lot of Huberman’s advice is stuff like “delay coffee consumption by 1 hour when you wake up” or “go to sleep in a cold room and wake up in a warm one”. You’re not taking on much risk by trying this stuff out. Admittedly, there are cases where this sort of thing leads people to very bad outcomes, but outside of those extreme cases I think you’re fine just giving it a go if you’re interested
I think there’s a larger harm from cultivating ecosystems of bad science that then overlaps into “the experts can’t be trusted” as seen with COVID.
Not to say that there aren’t legit issues with “experts” or that there aren’t pieces of interesting science. I think it’s tricky. And Huberman isn’t guilty by association for Joe Rogan’s misinformation in vaccines. But there are overlaps. And over-claiming because of bad epistemics (sometimes) is one of them.
@Joshua
I don’t know, on the one hand I agree with you, but on the other I think you and many of the other commenters are painting with too broad of a brush on Huberman. I agree with Andrew’s critiques in the original post, but I’ve never gotten the impression that Huberman is “anti-vax” or anything really close to that. He always comes off more like a Matthew Walker/Malcolm Gladwell/Ryan Holiday kind of guy if you get what I mean. I also don’t usually like going down the ‘this eventually leads to larger societal harm’ route, IMO this is a similar sort of error as the one in the ‘The Causal Revolution in Econometrics Has Gone Too Far’ post (sorry I don’t know if links are allowed). Ironically, of all the things I read/listen to online it’s probably this blog that has most diminished my respect for ‘experts’.
Blackthorne –
I think we are mostly in agreement here. Huberman isn’t responsible for the sins of others. And I’ve never heard him say anything anti-vax or even anything really similar to that either.
But there’s something off for me, that he goes on a Rogan show and plays footsies. It contributes to what I consider a materially harmful phenomenon. But I don’t know the solution. It’s not like I think Rogan shouldn’t be allowed to exist or even that Huberman necessarily shouldn’t take advantage of such an enormous platform. But it all makes me queasy.
+1
That’s directed to anonymous and mis-nested.
As I have often remarked, it is possible to learn a lot from reading this blog. Until today, I had never heard of Andrew Huberman but now I know he is quite prolific, professional and informative. And, in case anyone is interested in conceiving a child and/or living a long and happy life, I suggest this podcast of his
https://www.youtube.com/watch?v=ggQdPDPqv8k
in which he indicates what you should do with your cellphone. Apparently, a cellphone is one of the deadliest and most insidious devices known to mankind. Medically and not just socially.
Lets say cold showers might benefit some people. How about just try taking a cold shower if you want to? It is pretty easy for each individual to figure it out for themselves. What is the purpose of these studies?
I guess maybe some organization (military, schools, prisons) could use them in court to justify not paying for hot water.
Anyway, I like a hot shower with burst of cold at the end.
Anon:
Sure, but if a large and consistent effect really exists, then the authors are doing the world a favor by publicizing it, as that can motivate lots of people to try it out for themselves. On the other hand, if there really is no such strong evidence, then they could be causing damage by giving people unrealistic expectations. So the quality of the research on average effects is relevant.
Anoneuoid –
> What is the purpose of these studies?
I don’t understand your question. The answer seems obvious. The purpose is to get a picture of the probabilities related to benefit. There are cryotherapy places not far from me, is it worth spending money on? I’ve actually started to end my showers with a minute or two under cold water (our water is from a well, and very cold)… I kind of like it but I also want to kind of skip it. Having a sense of the probabilities related to potential benefit would be useful.
Is there a reason not to conduct these studies?
“How about just try taking a cold shower if you want to? ”
‘Uoid, we can’t have some people gaining an advantage from cold showers and creating social disparities. We need to know the truth about cold showers so we can decided if they should be banned or mandated and what exact procedures in showering should be banned and mandated.
Andrew’s joking aside, what we really need is more data, more data analysts and a new Department of Shower Disparity Prevention to analyze the data and assess whether everyone is getting equal benefits from showers.
> …What is the purpose of these studies?
The purpose of these studies is to win fame and fortune by exploiting the bottomless well of human desire for easy fixes. Same purpose as of lots of things.
When the military turn off the hot water at all the basic training camps, I will know for sure that this research is true. Until that day, I really don’t care too much. An added plus is that it will save some taxpayer money from lower utility bills.
Even the most convincing studies deserve an effort at duplication. I am going to take cold/hot showers based on a well shuffled deck of cards with red cards indicating a hot shower and a black card a cold shower. I am then going to rate sexual satisfaction on a one to ten score. My wife, fifty three years of marriage, is going to go along and also rate the performance. We should probably do this fifty times to get statistically significant results. I anticipate a preliminary report in 2087 or so.
https://www.theverge.com/2015/3/18/8225321/memory-research-flatworm-cannibalism-james-mcconnell-michael-levin is the worms story. Quite a tale.
LOL. It was only a matter of time before Haberman would attract the attention of statisticians like Andrew.
Haberman seems to be the Dr. Oz of the west coast and for neuro* stuff, monetizing the idea that scientific publications in that area deliver actionable things one can do RIGHT NOW to improve one’s dopamine levels and what not.
Some months ago, I looked at one of his “strong evidence” papers, and it appeared in Frontiers (itself a red flag for me) and had 5-6 subjects (unfortunately I forgot what the paper was) and was clearly an exploratory study. The way he talks about the published evidence, I seriously doubt that Haberman would be able to answer basic questions about what a significant effect tells you etc. In his telling, every paper he mentions provides definitive evidence, news you can use to optimize every little detail of your life.
Andrew, he also touts or touted athletic greens, I saw a web page calling it the Huberman Lab Supplement: https://www.vastdiversity.com/athletic-greens-huberman/
The antidote is the Gelman Lab Supplement. A healthy dose of Bayes, with skeptical priors.
Huberman’s success is basically an instance of monetization of basic science; my university also encourages profs to create companies and to monetize our work somehow. What’s really bad about the Huberman case is that he’s using (implicitly or explicitly) his Stanford branding to gain prominence (there was that other guy from Stanford too, name begins with a Z, who blamed young people’s problems to porn or something) and Stanford is quietly standing by letting him do it.
Huberman is a huge hit in the podcast world by the way. You should listen to his youtube podcasts or whatever they are called, they are hilarious. I think that the main thing I learned from the pandemic is that most people are just plain gullible. This accounts for his success.
Shravan:
I wonder what his colleagues at Stanford think about all this. I’d like to imagine they are upset about it, but who knows, maybe they enjoy the reflected glow of publicity. Or worse, from a scientific perspective, maybe they think he’s basically right about everything and has just fallen afoul of the statistical nitpickers. Kind of like the people who think that the Theranos team was great and the only mistake they made was to get caught.
A cautionary tale in this area is the story of the disgraced primatologist Marc Hauser who was kicked out of Harvard for misrepresenting his data, and then two of his most distinguished colleagues from the psychology department, Susan Carey and Steven Pinker, endorsed his hype-y followup venture. I would’ve thought they’d have felt angry and betrayed by Hauser, but I guess not, even though they were victims of the fraud, in the sense that Hauser had traded off Harvard’s reputation. I was reminded of those stories of religious leaders who commit crimes and then are defended by people from those religious groups, even though that’s where the victims of the crimes were coming from.
I blogged about this at the time: https://steamtraen.blogspot.com/2023/03/some-interesting-discoveries-in-shared.html
Interesting.
I’m left wondering about the implications of all those problems with the dataset. As you mention in the comments they don’t necessarily invalidate the study findings.
But in the end, I’d say it certainly suggests that the old rule of thumb should apply, and one study doesn’t mean much of anything let alone one study that hasn’t been replicated.
So then I get to that it’s not scientific to make much of this study. Huberman tends to caveat his material, but in balance I feel not nearly enough. There’s a kind of jnconsistency or contradiction in how he treats uncertainty. He explicitly caveats on a broad level but doesn’t carry those caveats through to the level of discussing individual studies or specific causes and effects.
It feels performative to me. Basically a leveraging of plausible deniability. I’m not saying it’s deliberate but it certainly is lucrative and attracts a lot of attention and a lot of fans and a lot of flattering comments from like-minded public facing science purveyors.
For all the problems of more traditional science-public interfaces – and there certainly are many – this does not strike me as an improvement and well may be worse.
But I wonder about my biases here, particularly because of many of the extensive political overlaps between Huberman and the IDW/contrarian types..
I used to be a big listener of Huberman, and I can tell you that a lot of the assumptions people are making about him in these comments are wildly incorrect. He is not a shill for supplements, and 99% of what he talks about is behavioral changes people can make to be healthier. He often mentions that supplements can be helpful, but can’t be a replacement for the basics. He mainly talks/focuses on things like getting enough sleep, sunlight, exercise, etc. As an anecdote, he is the reason I started waking up early and working out in the mornings, which has been a huge benefit to my life!
It’s a little sad to see people hate on him, especially because he is probably the most positively impactful public science educator of all time. Listen to a full podcast with him, he’s awesome!
PS: Also, I don’t understand the problem with the word “grit”. How is that not real?
I don’t know anything about Huberman so will reserve judgement but most of the issue here is that frequently the claimed effects in the kinds of studies being done on these topics are wildly inflated. Let’s say a cold shower actually reduces your risk of bad thing by 3%. That’d be good. But the study is likely to be done in a small sample and get a statistically significant reduction and because of the noise and the small sample the estimate will be wildly too big. Let’s say 27%. So now everyone is doing X to reduce bad thing by 27% but in fact only reducing bad thing by 3% and maybe increasing bad thing 2 in such a way that the net benefit is questionable… And of course spending cash time and effort they could have been spending on maybe walking their dog or shopping for more healthy food, stuff people are already aware of and on net has a bigger effect. This sort of wildly inaccurate quantification results overall in net harm by driving attention away from the known more effective interventions.
i want to see the raw data for this cold shower stuff.
I see your point. A science educator has a responsibility to not take studies at face value. But, as is often talked about on this blog, statistics is hard. Analyzing each study that comes out when you cover a huge number of topics is especially hard.
Axel –
Statistics is hard. But not taking any given study at face value, especially if it hasn’t been replicated, and. in particular if there aren’t other studies that have shown similar results, it easy.
Huberman, IMO, has a habit of making strong claims about large effect sizes, based on thin literature, quite regularly. Yes, he throws in general caveats but doesn’t seem to hold a consistent standard with respect to those caveats. For example, he might caveat a particular paper about something like cold immersion but then speak broadly about cold immersion benefits across multiple outcome measures as if they are established fact.
That seems to me to be a problem.
Joshua,
That makes sense. I don’t really remember from the podcasts I listened to, but I’m guessing you’re right about him overhyping effect sizes and such.
Axel –
He’s not as bad as some. And I think some of his pods are definitely interesting. And I don’t think he’s completely full of it. The main issue for me is that it’s part of a lager phenomenon and his stuff overlaps with stuff that’s quite a bit worse. As a general phenomenon it bothers me, and I think the net effect is pretty bad, with stuff like the dismissal of “experts” on public health issues like vaccines.
It’s all tricky. Huberman isn’t responsible for RFK’s nonsense when he goes in front of a massive audience like Joe Rogan’s. But the linkages, even if a bit tenuous, are troubling. I think we might be headed to a very bad place – say if there’s another pandemic and there’s a widespread dismissal of public health expertise largely as a result of the Rogan-types and the arms that reach out from that core.
OK, here comes a rant:
I’ve been on a bit of a jag listening to another podcaster I’d put in a somewhat similar category – even if he’s less irresponsible that Huberman – named Nick Jickomes:
https://www.nickjikomes.com/
Interesting podcasts. Interesting guests. But there’s something about it that leaves me feeling a bit queasy.
I’ll describe an example. He was interviewing a woman who’s done a bunch of research into olive oil and other oils that consumers buy. Apparently she did some pretty seminal work on olive oil about 15 years ago, where she empirically established that some 60% of olive oil you might buy in the supermarket was sub-standard either because it wasn’t actually 100% olive oil, or was heavily oxidized, or it was substandard “aesthetically.” (Interestingly, olive oil is apparently graded by a panel of 10 experts in what is considered a standardized and calibrated scale on what might seem like subjective evaluations like taste and color.).
Anyway, in her scientific research, she doesn’t typically identify the olive oils she conducts the research on by brand. Her perspective, which she admittedly struggled with clearly explaining, is that as a scientist she’s not in the business of rating olive oils by brands (she said she’s not there to police the industry) but to provide general information about the state of the olive oil market in the US. Jickomes kept pressing her on why she didn’t identify the sub-standard brands of olive oil and he eventually seemed to settle on a conspiratorial explanation that the UC (Davis) administration is in bed with the olive oil industry and therefore didn’t want UC researchers to besmirch olive oil companies that might contribute funding for research.
I mean I get his basic argument but I’m getting really sick of the facile conspiracy framework that seems to have become so prevalent. There are some legit reasons for the suspicions, IMO, but at some point there’s just a ton of unfalsifiable accusations and insinuations flying around and I think the net effect is deleterious. If you want to insinuate a conspiracy, do the work and make your accusation unfalsifiable. It’s just that basically anyone could imply that any research by anyone else is corrupted by a financial bias that’s mostly based on bad faith mind-probing.
Quite a rant that started with Huberman, and that’s not really fair to him. But I feel like all this stuff is pretty seamlessly connected, and lot of public facing science communicators and more general media figures and podcasters exploit that seamlessness on enormous platforms.
sorry – make your accusation falsifiable.
So many people going around making all these unfalsifiable accusations.
That has an easy fix: just don’t cover a huge number of topics. Bite off only what you can chew.
Shravan –
> That has an easy fix: just don’t cover a huge number of topics. Bite off only what you can chew
Yes. The notion of epistemic humidity comes to mind. It does seem rather basic, doesn’t it?
Ha. Epistemic humility also.
Shravan Vasishth: the problem is that the economics of pop culture commentating encourage people to respond quickly on a wide range of topics especially controversial ones. If you stick to your lane, you are likely to drop out of the public eye when the topic of the day changes or when everyone has read your book or seen your documentary that builds on 20 years of work. As the Russo-Ukrainian war reverted from maneuver warfare to positional warfare, it has been very clear how many foreign commentators saw their sources of independent information drying up and chose to become pundits rather than saying honestly “its hard to figure out what is happening, I still stand behind my general analysis from four months ago and I have nothing substantive to add.”
Money and fame can be addictive.
well, i am not a Stanford prof but i could have told you that exercise is good for you. i have heard his podcasts and they tend to be over the top. maybe you needed to hear from him that you should work out, but eg his dopamine life hacks sound just nonsensical to me. maybe i just don’t know anything about optimizing my life.
Axel:
You say “he is not a shill for supplements,” but what about the image from the above post: “Meet Rhodiola Rosea. A Powerful Adoptogen that Helps Combat Unwanted Stress and Fatigue,” etc.?
If that’s not shilling for supplements, I don’t know what is.
A fun anecdote. When I stayed at a major hotel in Stockholm in 2005, the end of the breakfast buffet line had a nice little basket of pills, one of which was ‘Rosenrot’. Being young and adventurous I didn’t ask too many questions and just took two of them! Turns out it was Rhodiola :) I do remember feeling a little boost, and have since then the couple times I’ve purchased fortified extracts of Rhodiola. IIRC there is at least a little bit of evidence, but honestly I’ve not looked into it, and it was more fun when it was just an esoteric herb that Scandinavian hotels thought their guests should have!
Andrew,
I’m guessing he didn’t write the website content. It also seems like we might have a different idea of what constitutes being a shill. If you listen to his actual podcast (at least when I listened to it), he does not try and overinflate the importance of supplements. He says behavioral and zero-cost methods of improving health are much more important. A shill would say that you NEED to take supplements to be healthy (what I would call a shill).
The best part of this study: the data are not actually available, despite what the journal page claims. I clicked on the link (on the journal website) that the data are supposed to be downloadable from, but I get a 404 error.
I knew I’d heard the name Huberman before, and doing a Twitter search determined last time was when Stuart Ritchie pointed out he was relying on small sample sizes and implicitly hacked p-values to push cold immersion as well as “psychedelics for mental illness, transgenerational epigenetic inheritance, “self-directed neuroplasticity”, and… making your eyesight better by looking at a pen(?!)”.
https://twitter.com/StuartJRitchie/status/1632411118260854784
To that I’d added that he’d promoted the work of a Harvard-affiliated doctor who was claiming the keto diet cures schizophrenia based not on any kind of control trial but instead just his patients (with one paper having a sample size of 6). This same doctor had been on the youtube channel of a woman with schizoaffective disorder, and while she at least didn’t talk about it afterward as comparable in effectiveness to actual anti-psychotic medication, I couldn’t help thinking that some more persuadable schizophrenic was going to wind up skinnier on the one hand (anti-psychotics are known to cause weight gain) but slipping into psychosis on the other.
Science should definitely be more rigorous than what it is being done right now in general. I really appreciate the discussion throughout this blog for training my statistics’ rigor, but it is also a bit sad to see we judge people with far less amount of rigor than we judge science.
Scientific claims should be considered false unless they passed several layers of evidence checks, but we don’t have to do this to humans…
It would be cool if Huberman hired a statistical consultant to assist for the show.
In fact, I know the perfect blog to post a job post :-)
Anon:
Unfortunately, Freakonomics no longer has a blog!
This isn’t good:
https://twitter.com/C_Kavanagh/status/1697010391082217639?s=20
While it’s commendable to see shitty science be taken apart, the need to lump Huberman in with the manosphere/Rogan/anti-vax just reeks of envy and pushback from those trapped in the walls of academia and its politics.
Anon:
Huberman’s gone on Rogan’s show and he’s also praised notorious anti-vaxxer RFK Jr. Perhaps this is Huberman’s envy about being trapped in the walls of academia etc.; nonetheless, he’s the one who’s doing this.
Everyone makes mistakes but even as someone not very stats literate, I’d suggest this Huberman fella maybe should train his mind a bit more before podcasting to millions based on his ability to interpret statistical inference from research:
https://x.com/Andercot/status/1788605009535655951
Or maybe I’m just envious.
I dunno, I looked at that clip and it makes more sense than a lot of Ted talks I’ve seen . . .
Hmmm. OK. It’s not a huge mistake and the reaction seems over the top. But still, when advising millions of woman as to when they should conclude “there’s something going on” if they aren’t yet pregnant, it seems it’s not to much to ask for some fact-checking.
Joshua:
The guy’s a bullshitter and will sling numbers around to try to make whatever point he wants to make. In that sense, the problem goes beyond fact-checking.
Shocking:
Retraction: Impact of cold exposure on life satisfaction and physical composition of soldiers
https://pubmed.ncbi.nlm.nih.gov/36599485/