Susie Neilson writes:
Tony Robbins was having a rough year. It was 2019 . . . Buzzfeed News was publishing a multi-part investigation into allegations that he had, during his in-person events, groped women and belittled abuse survivors. Robbins had a lot to lose. . . . he had amassed a net worth estimated at more than $400 million. His sweaty self-help seminars and grab-life-by-the-balls philosophy had spurred devotion from millions of followers, many of whom spent thousands on his products. . . . But at a precarious time, Robbins found an unlikely ally: Stanford.
No, not this guy. Not this person either, nor is it the hypester mentioned here. Not this group either. No, it was some other crew:
Around the same time that Buzzfeed News published its series, the Stanford Healthcare Innovation Lab, helmed by acclaimed genomicist Michael Snyder, launched a very different kind of investigation into Robbins’ seminars as part of an effort to identify “novel approaches to mental health.” In 2021, researchers affiliated with the lab, known as SHIL, published a study of “Unleash the Power Within,” Robbins’ four-day flagship seminar. . . .
Then, in 2022, SHIL-affiliated researchers — some of whom were fans and acolytes of Robbins’ work — published a more provocative paper. This one claimed that Robbins’ six-day, $4,500 “Date with Destiny” program eliminated symptoms of depression in 100% of initially depressed event-goers who were studied. In contrast, across clinical trials of antidepressants, just half of people report feeling better in six to eight weeks.
100% improved, huh? That sounds like big news! And, indeed, the Stanfordians were pretty proud, as well they should have been after such a clear and positive finding:
“This is going to be one of the most effective, if not the most effective, improvements in depression published,” Ariel Ganz, SHIL’s director of mental health innovation and the studies’ co-author, said in a 2021 video conversation with Snyder, other co-authors and Robbins.
In case you’re curious, the two papers in question are here and here
Sorry, now comes the bad news:
But when the Chronicle asked more than a dozen experts in psychology, statistics and medical research to review Stanford’s Date with Destiny study, many raised serious concerns about its validity. They found basic calculation errors, head-scratching data points and conflicting statements about how study participants were selected. Critically, they noted that too few people participated in the research for the findings to hold meaning for the public at large.
That’s a bummer when your study has data problems. Really too bad. That paper had 26 participants and 9 authors—that’s less than 3 data points per author, better than the student-faculty ratio at Ivy League schools. You’d think the authors could’ve avoided all these errors by divvying up the problem and looking carefully at the data from three participants each. Now they’re in the same category as that gremlins guy who approached the Platonic ideal of publishing a paper with more errors than data points.
The authors responded:
Snyder declined an interview, asking for reporters’ questions in writing. In response to a list of detailed questions, he acknowledged the study must now be corrected in light of the Chronicle’s findings, while noting that it was peer-reviewed. Snyder said that, in any event, the resulting paper suggested that “immersive interventions may be useful for reducing depressive symptoms and enhancing well-being.”
Ahhhh, the one-way street fallacy! Their data are also consistent with no effect, or with a negative effect.
Neilson continues:
Neither of the studies bore marks of research misconduct or fraud, according to the experts contacted by the Chronicle. Yet the work doesn’t appear to be of the caliber expected of a researcher with Snyder’s reputation, they said, or of a university such as Stanford. . . .
“Has he lost his mind?” asked Phyllis Gardner, Snyder’s colleague at Stanford’s School of Medicine, who was among the first to question the work of disgraced Theranos founder Elizabeth Holmes.
“That seems really surprising to me,” said Dr. Max Wintermark, the former deputy director of the Stanford Center for Precision Mental Health and Wellness, who now chairs the University of Texas MD Anderson Cancer Center’s neuroradiology department.
What was going on?? Here’s the story:
The Chronicle spent several months looking into Stanford’s unusual research. The partnership behind the studies formed when Ganz, then a new postdoctoral researcher in Snyder’s lab, met Benjamin Rolnik, a former Hollywood talent agent, at a wellness retreat hosted by one of Robbins’ close friends and collaborators, self-help guru Byron Katie.
Wow, this is a wonderful origin story. Much better than anything we have in statistics. The story continues:
Robbins and Snyder backed the same startup; they have also promoted each others’ products and business partners. . . . Robbins has also elevated controversial ideas and therapies. In the 1990s, he expressed doubt about the link between HIV and AIDS . . . Like Robbins, Katie has been criticized in online forums for belittling the experiences of abuse survivors during her own events. In 2012, she told an interviewer that she “never” felt sorry for people who experienced rape and other abuse, because “they only believe their thoughts” and were “perfectly all right.” In an email to the Chronicle, Katie said her statement had been “taken out of context.”
Wow. Also this:
Stanford did not respond to the Chronicle’s request for a list of donors to the genetics department and more information about SHIL members. . . . Under Snyder, Rolnik and Ganz went on to co-author four papers on Katie’s method, showing The Work could help stutterers, teachers and people whose genes put them at elevated risk for breast cancer.
And now some details:
James Heathers, a physiologist and affiliated researcher at Linnaeus University in Sweden, is best known for his work detecting errors in published research. . . . Heathers commended the Stanford researchers for publishing their data in a Dropbox folder, which he said is a good practice that demonstrates openness, but said he was left with broad concerns after reviewing it.
“Overall: This study is poorly conducted, and contains data handling errors, protocol violations, and other evidence of poor experimental practice,” Heathers wrote in a document summarizing his line-by-line review of the Date With Destiny study.
Critically, the researchers relied on patients’ PHQ-9 scores but calculated them incorrectly, Heathers noted. Instead of adding up each of the nine items, they added item No. 2 twice and skipped over item No. 3. These scores form the bedrock of findings about patients’ changing depression levels. “It is hard to imagine how this escaped inspection,” Heathers wrote.
OK, but this last bit is funny
I’ve tried to present the above story in an entertaining way, but ultimately it’s sad (people with health problems being led on by hype), infuriating (academics who are already rich and successful cutting corners to gain even more riches and fame . . . what is it they want, a jet ski made out of diamonds? Is there anything that would satisfy these people), and upsetting (as this takes up financial and attentional resources that could be spent on actual science).
But the story does have one funny part, and here it is:
Representatives for Elsevier, the company behind the Journal of Psychiatric Research, which published the study, declined to provide details about the article’s peer-review process, but said it “upholds the highest standards of quality and integrity across all journals.”
I have a horrible feeling they’re being honest there. This published paper with all these data problems and conflict-of-interest problems probably does match a standard of quality and integrity that occurs across Elsevier’s journals.
They’re not even pretending to be shocked that they published such a bad paper. You gotta hand it to them for being so brazen about it. Not.
I mean with an acronym like SHIL this group really just had to have this particular date with destiny didn’t they?
Ha, this was going to be my point too. Self fulfilling prophecy if there ever was one!
I am surprised that no one has pointed out the connections to the dishonest researchers who publish works regarding dishonesty behavior. I suppose there is some need to leave dangling clues which are obvious to the perpetrators and thus, elicit an extra bonus because they are undiscovered.
Paul:
We discussed that topic here and, in much more detail, here.
Is this really the one way street fallacy? The original example of the fallacy was failing to note that if a first child being a girl made parents conservative, then that meant the first child being a boy made parents liberal (if the two types of parents are being compared to one another). Here it seems that a weak or absent result is being interpreted as suggestive of a positive effect, when it could just as easily be compatible with no effect or a negative effect.
In both cases, there is a directionality to the blinders of the researchers/commenters, but there is a key difference: in the boy/girl case, there could be a strong and well-documented effect, but one half of the effect is being ignored; in the present case, there’s not much evidence of an effect at all. Andrew has often commented on cases like this one, where showing a result to be weak or absent leads to a proponent saying something like, “but the data is still suggestive of an effect”, when the data just as firmly suggest no effect or the opposite effect. There may already be a name for this fallacy, but if not, it needs one.
Gregory:
I’m referring to the author’s last-ditch effort, now that his strong claims have been discredited, to salvage something by saying that “immersive interventions may be useful for reducing depressive symptoms and enhancing well-being.” The author’s sentence is true—these interventions may be useful—but it’s misleading by not also including the possibility that the interventions may be harmful. And also excluding the more general possibility that the intervention can be useful in some cases and harmful in others, often with small effects. This seems like a one-way street fallacy—it’s a way to claim a positive effect by simply not thinking about the negative possibilities. As you say, I’ve talked about this a lot. To me, seems like an example of the one-way street fallacy, but if you can come up with a different name for this variant, that would be cool.
Andrew–
Yes, I agree completely about the last-ditch effort here, and that you’ve highlighted many other instances of the same. It’s just that the one-way fallacy would apply to cases where it was *not* a last-ditch effort to maintain that some result was useful– even a strong, obvious, robust, result could be misinterpreted (or, perhaps better, mis-emphasized) in the manner of the one-way street fallacy.
How about restricting one-way street fallacy to cases like the boy/girl-liberal/conservative case, and a new name for last-ditch attempts to save a result? The “last-ditch fallacy”? Or the “last-refuge” fallacy (especially if scoundrelism is suspected)?
If we distinguish the two fallacies, it is of course possible to fall prey to both at the same time– being blind to the street going the other way, and defending as “suggestive” a result consistent with no effects or effects opposite to the “suggestion”. You could have two fallacies in one case–a bargain!
Andrew is looking for a different name for the “one-way street fallacy.” According to Google Translate there are many ways to say this; here are a few to be used at appropriate moments:
falacia de la calle de un solo sentido
eenrichtingsverkeersmisvatting
erreur de la rue à sens unique
Einbahnstraßen-Irrtum
eyn-veg gas falasi
I must say that I had (and still have) a good laugh at the title, even before reading the post. The content is rather… depressing, reading about scientists selling their souls.
Raphael:
You see it as them selling their souls. My guess is that the bigshot scientists who do this sort of horrible thing see themselves as saviors—they’re Great Men who are going to cure cancer or whatever—and so any $ that flows to them will to do much more good than if it were to go anywhere else. Also some of these successful scientists spend time raising money for their labs by chatting up rich people (I’d do this too if I only knew how!!), and I expect than when you hang out with richies, you start thinking that you deserve the rich lifestyle too.
Beyond all this, science is broad. This dude could be an awesome biologist but have a confused understanding of the principles of statistical measurement and inference. I could believe that! After all, I understand the principles of statistical measurement and inference, but I don’t know any biology at all.
Also there’s a selection effect. There are lots and lots of sane, non-greedy Stanford biologists and sane, non-greedy Stanford medical school professors who don’t go around endorsing infomercials, cold showers, etc. The Stanford professors we hear about are often the ones who have a few screws loose.
In a recent post you wrote that university professors have three responsibilities: to educate students, to conduct research, and to serve the public. Aiding and abetting the motivational speaker industry is none of these things (I am not sure what else to call ‘writing a paper with too-good-to-be-true results claiming that my friend’s seminar helps with mental health issues’ other than aiding and abetting). I am not saying that no research should be done into the effects of motivational speaker seminars – far from it! But the poor quality of the paper raises the question of whether the researchers did the best they could. This is particularly bad when the researchers may be financially entangled with their research subject (they refused to answer this question, as noted above).
I should not have written about ‘selling their souls’, that is not for me to judge, independent of whether it is true or not. I wish I could take that back. I will, however, question how these researchers fulfilled their obligations to teach, research and serve by promoting a motivational speaker’s seminar.
Check out what’s happening at Stanford now: https://www.importantcontext.news/p/stanford-university-to-welcome-fringe
Just to flesh this out:
Yikes. It never ends.
Podcast about that conference. Here’s a preview: not a good look for Stanford.
https://www.conspirituality.net/episodes/229-stanford-has-fallen
The relevant part starts at around 12 minutes.
A main theme is the overlap between the public health focus of the conference with the economics focus of many of the presentations and organizers.
Why are places like Stanford and Johns Hopkins hosting gatherings of well-known coronavirus cranks?
https://www.thenation.com/article/society/stanford-covid-symposium-misinformation
Joshua:
Considering that something like 47% of voters plan to vote for Donald Trump, who endorses lots of conspiracy theories that are much more ridiculous than those of the coronavirus cranks, it makes sense that some nonzero percentages of Stanford and Johns Hopkins faculty would support such views. On the other hand, yeah, there are lots of minority views that still aren’t supported by conferences at these universities. For example, I’d guess that more than 10% of Stanford and Johns Hopkins faculty believe in astrology, but they’re not about to host conferences on the topic, as it would be just too embarrassing.
I think that if the coronavirus cranks did not have major political support, they wouldn’t be holding these conferences.
The other thing is that a lot of people are rightfully annoyed by the b.s. coming from mainstream coronavirus authorities, so they feel like it’s only fair to give the coronavirus cranks a hearing, as, for all their flaws, they’re providing an alternative perspective.
Andrew –
The political linkages are pretty important. I can’t shake the gut feeling that Trump’s going to win, and then there’s a good chance that some combo of RFK and the editor of the Epoch Times and decidedly rightwing figurehead (Vinay Prasad, Jay Battacharya, Marty Makary) and scientific outliers (Pierre Kory, Robert Malone) and bizarre online celebrities (Bret Weinstein, Jordan Peterson, Joe Rogan) are managing the levers of overlap between the state and scientific institutions? They’re going to want to entirely reshape the CDC and the NIH and the FDA, etc.
On the one hand I guess it could be seen as just the swing of the political pendulum. Left-leaning academics and researchers will be replaced by right-leaners. It’s not like all, or any of those institutions have been perfect. And clearly academia is largely stuck in an obsolete model. And so they won’t be perfect in the future just like they haven’t been in the past. And so now those institutions will reflect a different part of the electorate, who think vaccines cause autism and the mRNA vaccines caused 18 million deaths, and ivermectin was 100% effective against COVID, and cold plunges cure depression and obesity, and the pandemic was actually a “plandemic” where Bill Gates and Anthony Fauci were the planners.
On the other hand, I’ll admit it feels to me a bit like we’re doomed. I think much if not all of the “threat to our democracy” rhetoric is overwrought but the dissolving of what does or doesn’t comprise scientific evidence to be replaced by a WWE free-for-all feels a bit end of the world to me. RFK and Donald Trump/JD Vance as creators of public health policy during the next pandemic just can’t be good.
Joshua
I share all these sentiments. But I’d offer an amendment to your last sentence. “RFK and Donald Trump/JD Vance as creators of public health policy” could well turn out ok. It could even be better than what we have now, as long as “better” is defined as results and not process. In fact, much of the Trump support seems focused on the purported results of his administration – indeed, there is little good to be said about the process. That is what bothers me most about the changes you fear. I’m not sure that the results will be worse, but I’m fairly sure the process will be. A move away from truth and logic in public policy bothers me greatly, regardless of whether the results turn out to be a disaster or not. Too much political discourse is focused on the ends and too little on the means, in my opinion.
Dale –
I just happened to see this clip and I think it encapsulates the problem as I see it (and tying back to a previous discussion we were having with Anoneuoid). Looking at this I have a hard time visualizing how outcomes can’t possibly be anything positive when tied to such a broken process.
https://www.cnn.com/2024/10/06/politics/video/lara-trump-dana-bash-hurricane-misinformation-sotu-digvid
A discussion of FEMA and disaster relief is necessarily cast into a firehose of rhetoric about migration. What I see happening in the interface between science/research and government policy is a direct parallel. Pandemic public health policy necessarily becomes a discussion of liberal media bias and restriction of the “free speech” of conservatives and Bill Gates and one world government, even when Trump was the person overseeing the public health response.
Of course rightwingers see this in reverse and just a needed course correction. But I worry that public health policy and the institutions of science/policy development interface is going to turn into political rhetoric all the way down from process to outcomes (of course recognizing that someone like JD Vance says “same as it always was, only better”).
Consider that our very likely next vice-president and probably pretty likely future president is in record for laying out a very comprehensive political agenda based on a belief that the universities (and explicitly, also professors) are literally “the enemy.”
https://bryanalexander.org/politics/the-professors-are-the-enemy-j-d-vance-on-higher-education/
Joshua
I don’t have a hard time envisioning “good” outcomes from these bad processes. As much as I detest the statements and proposed policies – for example, regarding immigration – the result could easily be tightening the border against entry of criminals and terrorists while still enabling entry of others. Much depends on other branches of government as well as the extent to which the proposed policies are actually attempted (will they really try to deport 25 million people?). The one thing I feel confident in predicting is that the process regarding immigration will not build respect for human dignity and will not view immigrants as people but instead as tools that can add value to Americans (whoever that group comprises).
Another example might be climate policy. I don’t expect a Republican victory in November will do anything good for reducing global warming – and a Democratic victory would result in policies (at least proposals) that attempt to do something. Given the inefficiencies of any likely policies and the myriad indirect effects, my confidence in concrete conclusions is limited. What I am sure about is that a Republican victory will undermine logical thinking about climate change. I certainly fear what the outcome may be, but focusing on my fears is not that different than the Trump supporters focusing on their hopes. I believe that means and intentions matter, not just outcomes and results. My biggest fear is that the importance of means and intentions has mostly disappeared and is in real danger of vanishing completely from politics.
A prominent researcher does a study where there is a clear conflict of interest, comes up with such earth-shattering results, and allows it to be published full of readily detectable methodological problems. I can forgive mistakes in a study, but can the failure to scrutinize a 100% success rate for something as difficult as treating depression plus the conflict of interest leave room to trust any of the group’s other work?
Even if this is true, the author’s bias makes it hard to read.