“This is one of those views which are so absurd that only very learned men could possibly adopt them.” — Bertrand Russell.
In a comment on a recent post on the many failings of a published paper that purported to demonstrate a form of mind-body healing, Raghu Parthasarathy wrote:
Does anyone actually expect meaningful insight into the important topic of wound healing to come from a study like this?
The quick answer is: Yes, Raghu, many people appear to actually expect meaningful insight into the important topic of wound healing to come from a study like this! These people include:
– The editor of the journal, Scientific Reports (“an open access journal publishing original research from across all areas of the natural sciences, psychology, medicine and engineering. . . . the 5th most-cited journal in the world,” published by the respected Nature publishing group).
– The associate editor and reviewers of the paper, whoever they are.
– The psychology department at Harvard University.
– The two authors of the paper, both of whom are at Harvard.
– Harvard University itself, which publicized this work on its promotional magazine, the Harvard Gazette.
– Also Harvard when it released that statement a few years ago from a psychology professor and a political science professor that “the replication rate in psychology is quite high–indeed, it is statistically indistinguishable from 100%.”
– Celebrity economist Steven Levitt, who hyped this work on his Freakonomics podcast.
– Quasi-celebrity physicist Sean Carroll, who hyped it on his Mindscape podcast. Amusingly, the url for that podcast is preposterousuniverse.com. “Preposterous Universe,” indeed.
– The formerly-respected magazine Scientific American, which published a lay summary by the authors.
– The who-knows-how-many-thousands-of-people trust the Nature brand, Harvard, Levitt, Carroll, or Scientific American–along with the millions of people who’ve been “primed” (sorry!) by NPR, PNAS, Harvard, etc., over the past couple of decades to believe this sort of exercise in the creation and mining of noise.
So, Raghu, the answer to your question is, Yeah, it seems that a lot of people do actually expect meaningful insight into the important topic of wound healing to come from a study like this.
To actually expect meaningful insight into the important topic of wound healing to come from a study like this, all you have to do is take the word of Steven Levitt, Sean Carroll, the Nature publication group, Harvard University, and Scientific American.
These are the same sorts of sources that, just a few years ago, were extolling the “masterpieces” of the now-discredited food behavior researcher Brian Wansink.
Of all these people, the one I’m most annoyed by is . . .
Among all the people mentioned above who actually expect meaningful insight into the important topic of wound healing to come from a study like this, the one I’m most annoyed by is Sean Carroll.
Don’t get me wrong–it’s not personal. I’ve never met the man, nor have I ever corresponded with him. It’s just . . . he’s a physicist. A physicist should know better! I’m not saying that physicists can never do anything dumb–all of us have the capability for cluelessness–I’m just surprised and disappointed to see a physicist being dumb in this particular credulous way.
Why am I not so disappointed in the other believers listed above? For each of them I can see an . . . ummm, not an “excuse,” exactly, but a reason for them to actually expect meaningful insight into the important topic of wound healing to come from a study like this.
Let me go through them in order:
– The journal editor sees tons of submissions, probably only glances at some of them, and if you’re a journal editor you have to trust your staff, in the same way that, if you’re a football coach, you have to give the ball to the QB, no matter what you personally believe regarding his skills.
– The executives at the Nature publishing group want a continuing flow of papers and a continuing flow of income. They’ll outsource their scientific judgment to the journal editor.
– The associate editor and reviewers quite possibly work in the same field as the authors of the paper, in which case they may have been trained to actually expect meaningful insight into the important topic of wound healing to come from a study like this. Not good, but understandable.
– The leaders of the psychology department at Harvard University is going to promote the work of its students and faculty: they don’t see it as their job to judge the quality of the work, except for rare occasions such as dissertation defenses and tenure reviews.
– The two authors of the paper presumably came into the project with the expectation that meaningful insight into the important topic of wound healing could come from a study like this. Publication and publicity and recognition would just deepen their conviction on the topic, and it would be a rare research team indeed that would drastically revise their beliefs based on outside criticism. It can happen, but you wouldn’t expect it.
– The role of the Harvard Gazette is to publicize the university, not to vet the work they’re promoting. If you’re a writer or editor for that sort of publication, I guess you have to be a believer or else set your doubts at the door.
– Other Harvard faculty have an understandable loyalty to the institution, so whether or not they really believe that the replication rate in psychology is “statistically indistinguishable from 100%,” I can see that they’re not going to look hard for problems in work published by their colleagues at the university.
– Steven Levitt has published lots of credulous things over the years under the Freakonomics banner. He quite possibly thinks of this healing stuff as silly stuff–recreational social science, as is were–to which he would not apply the same critical standards to which he might hold serious work in economics.
– Scientific American . . . I guess they’ll publish pretty much anything nowadays. They probably weren’t vetting the science of the article they published–after all, its authors came from Harvard!
– The thousands of trusting readers . . . what can I say? You have to put your trust somewhere. As noted above, decades of promotion of crap science in academia, the news media, and, more recently, social media (as discussed here) have numbed the audience.
So, we’re left with Sean Carroll. He’s not at Harvard, nor is he a psychologist, so he has no reason for institutional loyalty here. According to his website, his current interests include “foundational questions in quantum mechanics, spacetime, statistical mechanics, complexity, and cosmology, with occasional dabblings elsewhere.”
Physicists do very careful experiments, and they have very precise theories. So I’d expect them to be skeptical of the sorts of vague theories and sloppy experiments as in the above-discussed paper. But also I’m reminded of the stories that Michael Weissman has shared with me about the physics establishment endorsing really bad physics education research.
Which makes me wonder if many physicists, including Carroll, just have a very naive understanding of how social science works? Maybe they have an (unjustified) belief in the rigor of randomized experiments and hypothesis tests. Statistics textbooks are always slamming research that doesn’t involve random sampling or randomized treatment assignment, and they go on and on about how to calculate p-values, while talking very little or not at all about substantive theory and measurement. I could imagine an uninformed outsider such as Carroll getting the vague impression that randomization plus statistical significance equals rigorous science. Which would be scary, if that’s the reason.
(On the flip side, one thing I’ve seen from some mathematicians and physicists is a naivety in the opposite direction, for example saying that all observational social science statistics is wrong because there’s no randomization, or saying that all polls are bad unless they use pure random sampling, etc.)
Another explanation for Carroll’s promotion of this particular bit of junk science is a simple celebrities-stick-together thing. When an academic celebrity hears about another academic celebrity, there’s the temptation to do some backscratching: everybody wins, right? The half-fame of moderate academic celebrity is maybe more pleasant than dealing with annoying academic things such as faculty meetings, grading, and getting papers rejected by journals), and if you maintain that by giving softball interviews to other academic celebrities, that could be a small price to pay.
I’m still guessing that Carroll is not as gullible has he appears to be in that podcast, that if we were to put him on the spot he’d say something mellow like, “Hey, sure, I know it might be junk! But if it’s junk, it won’t replicate. That’s the self-correcting nature of science! I’m just open-minded and writing about fun stuff.” To which I’d reply: sure, that skill–being credulous and publicizing press releases and giving softball interviews to frauds–is a skill that 80 zillion NPR reporters and Ted talk executives already have. You have a Ph.D. in physics, and that’s how you want to spend your time?? Just seems weird to me. But I guess that it’s a lifestyle that some academics enjoy, and to get there, it helps to tell people what they want to hear.
Again, this is nothing personal about Sean Carroll, a man I’ve never met. He’s just an example of a larger phenomenon of interest, coming from Raghu’s seemingly-simple question, “Does anyone actually expect meaningful insight into the important topic of wound healing to come from a study like this?”
P.S. In comments, Raghu writes:
Your statement, “Which makes me wonder if many physicists, including Carroll, just have a very naive understanding of how social science works?” Yes. It may be naive to assume studies reported in journals are done well, but it’s somewhat understandable. . . .
Scientists from more rigorous fields — yes, a loaded and overly general word, but I’m writing quickly — tend to assume that quantitative (i.e. there are graphs) studies in other fields are also rigorous. It does not occur to many people that they are not, readers of your blog excepted. It is genuinely shocking to many scientists from more rigorous fields to encounter work in fields that are full of noisy, p-hacked, junk published in “reputable” journals. I’ve seen this repeatedly with physicists encountering physics education research (some good stuff, a lot of awful stuff), microbiome research (my own field), and more. . . .
Interesting. Still, you’d think the specter of a psychologist claiming to have evidence for mind-body healing would trigger a physicists’ skepticism, no? Or maybe the physicist would have a vague feeling that, yes, nature is mysterious, so why not? Kind of like how in the 1970s there were physicists who believed there was evidence for extra-sensory perception.
Your take on a physicist should know better reminds me of the quote from the Lord of the Rings: “Treebeard: [after seeing the torn-down forest around Isengard] Saruman! A wizard should know better!” Seriously, you are entitled to select any of the cited people/organizations as bothering you the most, but if the criterion is who does the most damage, I suspect physicist is low on the list. The others have far more influence than a physicist.
But your rationales for each actor’s expectation that meaningful insights could come from a study like this seem a bit superficial to me. All of the listed reasons are real – but I think they all reflect deeper realities. Modern life seems to require quick astounding experiences. This shows up in media habits, leisure pursuits (I believe recreational activities like backpacking are declining in popularity compared with high intensity activities – I could be wrong, but I believe I’ve seen evidence related to this that I can’t put my fingers on), and our increasing reliance and trust in computers and automation to govern our lives. I’m not saying these things are bad per se – but I do think modern life is rapidly changing in fundamental ways that make it natural to believe that small undetected influences (such as timing and healing) can have large effects. In academia, the idea of a lengthy careful research agenda seems to be paling in comparison with quick attention-demanding results. In that sense, all of the actors you list have a lot invested in these beliefs. So does the general public. If we question that such small things can have such large impacts, then much of our current life experience is also called into question.
Dale:
I completely agree that, of all the people and institutions here, the physicist is probably doing the least damage. After all, who cares what some dude says on an obscure podcast. When it comes to responsibility, Harvard is arguably doing the most damage, also Freakonomics is doing damage given its long track record of promoting junk science. And I’m certainly not trying to liken Carroll to Saruman!
I’m most annoyed by the physicist, not because I think he’s doing the most damage but because I feel he has the least excuse for getting conned by this mind-body healing b.s. I feel like he has to actively work on getting conned. For the other actors in the drama, getting conned by b.s. is just something that happens to them all the time.
Admittedly, the last time I was interested in football, Bill Swiacki was a star at Columbia University, and Red Grange was sagaciously informing Ted Husing that a “six three two one defense was a hard one to run against,” but is the following now a true requirement?
“if you’re a football coach, you have to give the ball to the QB, no matter what you personally believe regarding his skills.”
Paul: it is a practical requirement, not a real one. There are, from time to time, teams that run a so-called “wildcat formation” in which the QB does not receive the center snap, but it is a rare event, and long removed from the days of the single wing, in which theQB was not routinely the first ball recipient.
Furthermore, in the last forty years, the unmistakeable trend in football is reduction of running and an increase in passing, in which the QB is preactically, if not always, involved.
Columbia, by the way, achieved their first share of an Ivy League football title since 1961, and only their second since the formation of the Ivy League in 1856. Maybe it’s time to get involved again.
ummm…. 1956.
Even before my time
https://vimeo.com/120703272
Video of 1934 Rose Bowl – Columbia 6 Stanford 0 Original Pathe News film
“The Rose Bowl’s greatest upset came on Columbia’s hidden-ball trick in 1934”
—————————————————————————————
Things soured as the years went by, but picked up briefly in 1988:
Victory Celebration Ending 44-Game Losing Streak | Oct. 8, 1988
With a 16-13 Homecoming win over Princeton, Columbia saw its 44-game losing streak come to an end. Princeton misses short on a 49-yard field goal attempt on the game’s final play and the Lions erupt in celebration. Solomon Johnson’s 2-yard touchdown run with under five minutes to play seals the win. Columbia’s Greg Abbruzzesse rushes for 182 yards on 37 carries. The victory also marks Columbia’s first-ever win at Lawrence A. Wien Stadium.
You’d be shocked how many celebrity physicists have never had a new idea empirically verified.
Also a random related note: Pierre and Marie Curie investigated mystics at seances after winning the Nobel physics prize together.
“[…] I’ve seen from some mathematicians and physicists is a naivety in the opposite direction, for example saying that all observational social science statistics is wrong because there’s no randomization, or saying that all polls are bad unless they use pure random sampling, etc.”
These arguments drive me crazy. I know a mathematician who likes to talk about his “fundamental critique of the survey.” His critique boils down to this: because there will never be a truly representative survey, all conclusions drawn from a survey are invalid.
Like, whaaaat?! Surely you can estimate the resulting bias, hypothesize the mechanism leading to non-response (or whatever issue arises), and adjust the responses for known covariates! I even sent him your example (I think it’s in Regression and Other Stories) where you draw meaningful inferences from a highly selective sample of Xbox Live players.
And his response? He sent me a mathematical article about random sampling. Aaahhhhh!!!
> Also Harvard when it released that statement a few years ago from a psychology professor and a political science professor that “the replication rate in psychology is quite high–indeed, it is statistically indistinguishable from 100%.”
Still hoping to read a debate between you and Gary King on this topic. This is one of the few things, that I am aware of, about which the two of you disagree.
I’m also inclined to expect a physicist to know better, but I doubt experience would prove that out. Seems like academic scientists from many disciplines were drawn to all kinds of nuttery back in the 60s & 70s. Unfortunately, education doesn’t seem to be the antidote to nutty beliefs. If it seems like fewer physicists are drawn to such things, it’s likely only because there are so many fewer physicists.
Education is not an antidote to nutty beliefs, but it should at least be a nice buffer to credulously believing the nutty beliefs of others.
I’ll try to briefly defend my incredulity.
Andrew writes, “many people appear to actually expect meaningful insight … to come from a study like this!”
“Appear” is important here. I’d say (agreeing with most of Andrew’s post) there are:
(a) people who think awful studies are good, and (b) people who don’t think awful studies are good, but act as if they do.
In group (a), there are (a1) people who are incompentent, perhaps because of poor training, who can’t assess quality, and (a2) people who don’t assess quality and assume the awful studies are done well.
Group (a2) is the interesting one. I can’t speak for Sean Carroll (though in grad school I took two classes he taught!), but I’m more sympathetic to this perspective than you are, Andrew. Scientists from more rigorous fields — yes, a loaded and overly general word, but I’m writing quickly — tend to assume that quantitative (i.e. there are graphs) studies in other fields are also rigorous. It does not occur to many people that they are not, readers of your blog excepted. It is genuinely shocking to many scientists from more rigorous fields to encounter work in fields that are full of noisy, p-hacked, junk published in “reputable” journals. I’ve seen this repeatedly with physicists encountering physics education research (some good stuff, a lot of awful stuff), microbiome research (my own field), and more. In fairness, physics has problems of its own — unverifiable stuff, trivial and totally uninteresting stuff, etc. Returning to the point: Your statement, “Which makes me wonder if many physicists, including Carroll, just have a very naive understanding of how social science works?” Yes. It may be naive to assume studies reported in journals are done well, but it’s somewhat understandable.
Maybe I’ll find time to write more later, about the other categories.
Raghu:
I think you’re missing the largest category: people that don’t understand rational thought at all, much less how to solve a problem using it. Many people “solve problems” – or think they are solving a problem – with social knowledge. Like the guy I worked with who, whenever the project manager was scheduled to show up at our jobsite, tried to think up questions for the project manager that would make him look smart. It never occurred to him that he might look really smart if he was actually trying to solve some of the problems on our project and came about his questions in the process of actually trying to do his job.
Another example I’ll never forget: studying with other students on a rare occasion, I realized they completely misunderstood some problem. I explained to them how the process worked and where they were misunderstanding the process. Zero uptake. Their response was “Dr. Johnson said….” OK, well I don’t know if he really said that or not, but if you think through the process there’s no way that’s correct. They didn’t seem to have any actual understanding of the process at all. I guess they had some contorted model in their mind based on their misunderstanding of the prof’s words. But following it through step by step it obviously didn’t make sense.
We could chalk the COVID disaster up to a similar misunderstanding. Five or six early CDC reports indicated quite clearly that it was transfered via aerosol – at least to peole who understand how things work. The evidence was highly consistent with the earliest phase of the 1918 pandemic and several experts got this as well (eg. Michael Osterholm), but the bungling “health care” bureacracy took over and enacted their misunderstanding of what the their peferssers told them.
Perhaps the reason education seems to be not doing so well is that, in our increasingly social environment, where individual knowledge has declining relevance, people just find it easier to do what their friends say or think, right or wrong, than to learn and think for themselves.
> Physicists do very careful experiments, and they have very precise theories. So I’d expect them to be skeptical of the sorts of vague theories and sloppy experiments as in the above-discussed paper.
I’m a physicist (theory, like Carroll). There is a strong separation between theory and experiment, only chemistry has something similar, I think. Theorists know almost nothing about statistics or non-physics experiments. Even the details of most physics experiments are not taught to us, only the basics (maybe unless in the course of your research you work with experimentalists on interpreting the results). An experimental physicist would likely have pushed much harder on the statistics and experimental design.
> Which makes me wonder if many physicists, including Carroll, just have a very naive understanding of how social science works?
Yes. I agree with Raghu when he says “Scientists from more rigorous fields…tend to assume that quantitative (i.e. there are graphs) studies in other fields are also rigorous”.
Though there is the physicist arrogance streak, kind of like what you mentioned. Physicists are usually amenable to arguments that social science is methodologically terrible, once it is explained.
Feynman was well aware of the problems in social science in the 1960s-70s. He talks a little bit about it when he talks about a series of well-controlled rat experiments and then says that it was only one guy who did these good controls and none of the experiments after that incorporated them. If only more of us could be like him…
Just another illustration of the fact that rigor in one field does not automatically translate to rigor in another.
Anon:
There’s also selection bias. I can only assume that most physicists in the 1970s were skeptical about extra-sensory perception; the few who weren’t got the attention. Similarly, I’d bet that most physicists today would be skeptical about big claims for mind-body healing–but the one who’s a believer is the one who gets the media exposure. I’m not saying that Carroll entertains the belief as a way of getting press; rather, a willingness to go against conventional wisdom (in this case, by actively avoiding skepticism about a dubious theory) is part of a bundle of traits that are be helpful in getting media connections.
The other thing is something we’ve discussed before on this blog, which is that when academic researchers get media attention for their work, it’s typically universally positive. So it’s natural for them to keep the positivity vibe going and not be a party pooper.
The Rebbe said it is better to be right than to be smart, and better to be kind than to be right.
You lost me at “mind-body” healing. There is no mind AND body – there is only body. So it should read “body-body” healing. Seems much more reasonable, doesn’t it?
Note: I didn’t read the study.
Lurking:
The abstract of the paper refers to “the theory of mind–body unity—which posits simultaneous and bidirectional influences of mind on body and body on mind.”
“Bidirectional,” huh? That sounds pretty erudite. But, hey, they’re at Harvard. At Harvard you’re allowed to be sophisticated.
I think there is something to the perception of pain and quality of life changing our perception of what subsequently constitutes ‘healed’ – e.g., if you take a positive outlook on an injury, you might perceive the long-term effects of the injury less. I will say though this is an overly generous interpretation of the original study.
There are other physiological mechanisms that mediate the relationship between a positive outlook and actual healing or risk of long-term sequelae. For example, that ‘positive outlook’ may reflect adaptive coping, leading to a lower or shorter-period of sustained stress, which in turn results in better immunological functioning.
I’ll confess being more than a tad prejudiced against Sean Carrol, as he’s a String Theory appologist* who gets some amount of flak over at Not Even Wrong, but there’s a far worse example. See this book review:
https://scottaaronson.blog/?p=7321
There is also the phenomenon of the famous physicist who thinks he’s smart (it’s always a he) and says something stupid about another field…
But don’t listen to me: I was dropped on my head by a phycisist when I was a baby (really!).
As I’ve said before, as I was heading off to MIT to study comp. sci., a seriously famous physicist (Manhattan Project no less), visiting my father’s business took me aside and said “Son, you can study anything you want, but just remember, if you are not a good physicist, you aren’t a good human being.” (Really!). My last band as a violinist (I put a string quartet together to read the Dvorak quartets) had one of Heisenberg’s kids on cello. He was actually a pretty kewl bloke.
*: Here’s one of the fathers of string theory fessing up to (1) String Theory as it exists today can’t possibly describe the real world and (b) all attempts to extend it beyond those contraints have failed. But (3) it’s still worth doing.
https://www.youtube.com/watch?v=_76J8_Wx91g
Go figure.
David:
Sure, this Kaku guy sounds like a charlatan–if he didn’t have that physics degree, he’d be the modern-day equivalent of the Chariots of the Gods guy or the Jupiter Effect guy, but he has that physics degree so he’s a charlatan with a physics degree. His motivation is clear: he can exaggerate or even just make things up, whatever it takes to keep pumping out books that will keep him on the lecture circuit. (It says here that his speaker fee is “$50,001 and above.” Kinda makes you wonder what would happen if you offered him $50,000.99.) I’d like to think that writing bullshit would decrease his value as a public speaker, but I guess that’s counterbalanced by the publicity benefits of coming out with a new book, which, I see from Amazon, got enthusiastic reviews from the New York Times and elsewhere.
Carroll, though, somehow he seems more sincere. It’s hard for me to believe that he really believes that mind-body-healing crap, but it does seem that he put a lot of effort into avoiding thinking about the topic at all. That whole podcast, he’s working his ass off to keep his mind empty of any critical thoughts, kind of like how if you’re underwater you can keep breathing out air in order to stop any water from getting in.
When you are underwater, you should breathe out. Years ago I read that the sensation of drowning is not caused by a lack of oxygen, but by elevated levels of carbon dioxide. I can attest to the fact that you cannot dive very far without exhaling. Diving without exhaling just feels awful.
Do you know if there was a live cat inside the cello?
Yes and no.
Maybe it was not the wound healing that interested Carroll, but rather the frumious clock. Anyone who has done a road trip with children knows that subjective time is more important than objective time. And clocks are in the physicist’s bailiwick.
I haven’t read the paper but I’m guessing that a lot of the people mentioned in the post didn’t either. At that point all you have is a press release and your pre-existing knowledge, and I think there’s a population of people who think their body is pretty mysterious. In that broad sense, and with Raghu’s idea of an assumption of good work, I can see why some people would buy into one particular research finding.
Two related memories: a classmate in high school saying “if my body had a Krebs cycle, I think I’d know about it” (he was joking but there’s a hint of truth there). The other is going to childbirth classes with my wife and learning about all the ways that a pregnant body changes to accommodate and grow a baby. It sounded so contrary to normal bodily function that it couldn’t be true, but of course it was.
Alex:
The scary thing is that, even if Carroll, etc., had read the published paper by Aungle and Langer, I’m guessing they still would’ve believed it, in the same way that the journal editors and referees–and, indeed, the authors themselves–were fooled. That’s the big problem, that it’s so easy for people to conduct an experiment and write a paper in a way that appears there is strong evidence for their theory, even when no actual strong evidence is presented. And that was why Nick and I wrote that paper, to demonstrate in this one example how it is possible to peel back the logic and figure out what’s going on, and also to point to some particular issues regarding data analysis and literature review. Again, these would be tough for outsiders–or insiders–to notice. The first problem was a subtlety regarding hierarchical models, and the second problem involved looking carefully at the cited literature.
So, from that perspective, Carroll’s real problem was not so much in trusting that particular study but rather in thinking that, just because a paper came out of Harvard and was published in a peer-reviewed journal, that it could be trusted. Also, he interviewed Langer, who is a confident person in an interview and is willing to say things that are not true. I’ll put that one on Carroll: as a host of a podcast, he should know better than to just trust whatever people are saying to him.