The other day we discussed some hype around the article, “Impact of cold exposure on life satisfaction and physical composition of soldiers,” published in the journal BMJ Military Health. According to a Stanford professor, podcast participant, and supplement salesman, “deliberate cold exposure is great training for the mind.”
But some have expressed skepticism regarding that study, with, as our correspondent Matt Bogard put it, “n = 49 split into treatment and control groups for these outcomes (also making gender subgroup comparisons).” There are times when n=49, or even n=1, can be enough, but not when estimating the effects of subtle treatments on highly variable outcomes.
In comments, Shravan Vasishth points out that the data should be available from the journal website. And, indeed, here’s the link, https://militaryhealth.bmj.com/content/early/2023/01/03/military-2022-002237.long:
Scroll down and you’ll see this:
Take that, you haters! You can click on the link, aaaaand:
OK . . . so let’s check the Internet Archive. The page is https://web.archive.org/web/20230000000000*/https://www.vyzkumodolnosti.cz/en/datasets, and here’s what we see:
So, 7, 12, and 14 Mar 2023. Clicking on any of these yields the following:
The first four links work, giving spreadsheets that appear to be raw data! The last two links give nothing; they just point back to this page.
For reasons discussed in my earlier post, I don’t have much interest in these data myself, but, for anyone who’s interested, just follow those links at the Internet Archive.
I am always super interested in such data so i will look at it. It is funny that the journal is unable to publish such an important url. This has happened to me too in Glossa psycholinguitics, there the journal says they are understaffed so they have been unable to fix it two months after publishing.
The third and fourth seem fine, but the first two have formatting issues that could make them challenging to analyze.
Link rot is a thing, but it’s disconcerting that the data went missing after less than a year.
I still have the data. ;-)
See https://steamtraen.blogspot.com/2023/03/some-interesting-discoveries-in-shared.html who found some other interesting stuff about the study
Ooof…I was willing to grant some (marginal) merit to the data, but between this and the disappearing data files, it has so many issues I don’t trust it at all, even as a small-n, p-hacked study.
OMG, Nick Brown’s deconstruction could be made into a three-season Netflix comedy series: Adventures in Data Science. I can totally imagine Ionesco-esque scenes playing out (sorry, I am not a screenwriter or playwright, but you get the idea):
Author1: “Hmm, this subject did the study in March, but why not change his record to show that he did it in January?”
Author 2: “Yeah, sounds like an excellent idea.”
Author 3: “By the way, is there any reason why we can’t be subjects in the study?”
Author 1: “Gosh, that is a fantastic idea. We even know what we want the data to show.”
Author 2: “Also, we can just duplicate some subjects. This will save us money and time.”
etc.
More seriously, it is just embarrassing that people like Huberman superficially glance at substandard papers and then proceed to make recommendations to millions of people on how to improve their lives. It’s great that his advice includes things like, get lots of sleep, exercise more. He should limit himself to the obvious stuff, because then even if the studies he cites are crap, the advice will still have positive practical benefits for his audience.
Actually, Nick Brown himself would be the best person to write this Netflix series. He has a wonderful knack for gobsmack-inducing understatement:
“In view of this, it seems difficult to be certain about the actual sample size of the final (two-condition) study, as reported in the article.”
“this” refers to identical subjects being assigned to both the control and cold treatment groups.
Thanks for your support! (I will add a comment to the blog post to this thread to show that this is indeed me.)