From the team that brought you “good-looking parents are 36% more likely to have a baby daughter as their first child than a baby son” and “The PDO cool mode has replaced the warm mode in the Pacific Ocean, virtually assuring us of about 30 years of global cooling” (background here and here) comes a new nugget of 24-carat credulity:
I [economist and author Steven Levitt] cannot think of an academic whose research findings have more consistently surprised me than my guest today, Harvard psychologist Ellen Langer. She’s a scientist, but her results seriously challenge the beliefs of mainstream science.
If the findings consistently surprise you, and they seriously challenge the beliefs of mainstream science, then maybe you should more seriously consider the possibility that these findings are wrong! Langer’s much-publicized work has been questioned before (for example, here and here).
Levitt continues:
I’ve got a model in my head of how the world works — a broad framework for making sense of the world around me. I’m sure you’ve got one, too. My model is, I think, pretty typical of someone who puts faith in modern science. Perhaps with a little added cynicism about human nature. So when I hear about a new research study, I have a habit of asking myself, “Given my model of the world, what results would I expect the study to generate?” Usually I’m pretty good at guessing what the researchers actually find. But with Ellen Langer, over and over and over, she gets results that I would never predict. So here are my questions for you as you listen to this conversation: First, do you find her research results as stunning as I do? And the second question I’d like you to think about is when research findings surprise you, what’s the right reaction? How do you know whether you should believe surprising results?
Good question. Excellent question. Indeed, it’s a question that often comes up with research that’s been promoted on Freakonomics.
So let’s see how Levitt handles his own question. He’s interviewing Langer on his podcast:
LEVITT: I’ve read the work of many scholars and I can honestly say that you win the prize for the body of research that most consistently finds results that are completely the opposite of what I would have predicted. You and I have completely different models of how the world works. And the data keep supporting your model. . . .
That’s not true! The data don’t keep supporting Langer’s model. Just for starters, see the two links given above, or this discussion by linguist Mark Liberman from 2009). Or my recent paper with Nick Brown. Or this recent post at The Skeptic.
This is a big, big problem. If you come into the discussion with the question, “How do you know whether you should believe surprising results?”, then you can’t just say, “the data keep supporting your model.” By making this unsupported claim about the data, you’ve given up your investigation before you’ve even begun!
Let’s go back to the interview:
LEVITT: Now, my first reaction to hearing the findings of your counterclockwise study would’ve been, “Well, that result will never be replicated. It has to be a fluke.” But it actually has been replicated a number of times, right?
LANGER: Yeah, and in different ways. I mean, to me, the important thing was the test of the mind-body unity idea. The next study in that series was a study Alia Crum and I did where we took chambermaids. And first thing, we just ask them how much exercise they get. And surprisingly, they don’t think they get any exercise because they think exercise is what you’re supposed to do after work. And after work, they’re just too tired. So for the study, what we did was very simple: we just taught half of them that their work was exercise. Different things that they’re doing — making beds, cleaning the windows, and what have you — were likened to working in different machines at the gym. And so at the end of this we found that this group wasn’t working any harder, eating any differently. Everything was basically the same as the group that wasn’t taught this change in mindset.
LEVITT: So there’s no intervention other than teaching.
LANGER: Exactly.
LEVITT: Nothing can possibly happen.
This is wrong in several ways. First, Levitt starts out by accepting that a certain suspect claim “actually has been replicated a number of times.” Going with your interviewee can make sense in a podcast, but, again, it’s counter to Levitt’s earlier goal of asking, “How do you know whether you should believe surprising results?”
Second, Levitt accepts without question Langer’s claim that “at the end of this we found that this group wasn’t working any harder, eating any differently. Everything was basically the same as the group that wasn’t taught this change in mindset.” As Brown and I discuss in our above-linked paper, that claimn of no changes in behavior is not supported by the data. The paper in question does not report any direct measures of diet and physical activity at either the start or end of the study, just information from a retrospective questionnaire. It is problematic to take survey responses as measures of actual behavior, especially in the context of a study of an intervention specifically designed to alter perceptions of exercise. Beyond this, the data in the study actually do show a large increase in perceived amount of exercise (the average going from 3.8 to 5.7 on a 0–10 scale).
OK, we’re getting into the weeds here, and I can hardly expect for Levitt to have read an unpublished paper on my website in preparation for doing this podcast. The larger point here is that he should be skeptical. He’s interviewing someone who’s produced a steady flow of counterintuitive—many would say implausible—claims, that have been much disputed. So when Langer makes a statement about one of her studies, sure, let her say her piece, but then engage your skepticism. Don’t just take her words at face value.
When Levitt states, “Nothing can possibly happen,” he’s implicitly endorsing Langer’s unsupported claims. It would be better for him to say something like, “OK, if what you say is true, that there were no behavioral changes, then nothing should be happening.” Sure, it’s just a podcast, he’s reacting in real time, and everyone makes mistakes—but, again, if your goal is to ask, “How do you know whether you should believe surprising results?”, then at some point you have to put on your skeptical hat.
There’s a lot more interview, then near the end we finally get to some pushback:
LEVITT: There’s a concept of mindfulness and we’ve talked about that. . . . And it’s completely implausible that simply by thinking or believing something different, you could make pain go away. It’s interesting that for you, those two are the same thing, but for me, they’re completely different. . .
Langer gives a long response (you can read it, it’s all in the transcript), Levitt keeps the conversation going, and then we get this:
LEVITT: I’m skeptical, at some level, of many of the things that you profess.
LANGER: Don’t say that!
LEVITT: Although sympathetic and hopeful. But what’s interesting as I just reflect, the people I know who are in their 80s and their 90s, who have influenced me the most, it is my father, my grandfather, Danny Kahneman, a few others — they all exhibit the exact features of mindfulness and aliveness that you’re describing. And I’m definitely going to add you to that list because in many ways, the most convincing thing to me about this conversation is that you embody the ideas you talk about. You are obviously mindful, and as you talk, it is exciting to talk to you and that enthusiasm feeds into my own experience. I’ll walk out of this discussion and for at least a few hours, hopefully longer, I’m going to be more mindful.
What kind of skepticism is this, Steve? You say you’re skeptical and then you immediately fold!
Levitt then summarizes:
How do you know whether to believe surprising results? I’m still grappling with that second question myself. I would love to live in the world suggested by Ellen Langer’s research. A world where I could will pain away simply through the power of thinking. A world where I could control aging gracefully. But it isn’t so easy to toss out a life’s worth of believing that those things just aren’t possible. So my own reaction to Ellen’s research is to be open minded to the possibility she’s right, and I’m actually going to spend some time practicing her approach to mindfulness and also going forward, I’m going to be very attentive to any evidence I see in my own life that supports her worldview. I haven’t really looked for that kind of evidence because I didn’t believe it was possible. But let me be honest with you, I’m a lot more open to Ellen’s research because I like the findings.
It’s good to be open. But it’s a big, big mistake to take suspect claims at face value. The moment Levitt said, “the data keep supporting your model,” he was already gone, game over.
Previously on Freakonomics
Nudges by Chopstick: From 2009, a completely uncritical plug of the work of Brian “Pizzagate” Wansink.
There’s No Such Thing as a Free Appetizer: From 2014, a completely uncritical plug of the work of Brian “Pizzagate” Wansink.
A bunch of completely uncritical plugs of Dan “Shreddergate” Ariely.
Are Cornell Students Psychic?: From 2010, a completely uncritical report of the notorious ESP experiments by Daryl Bem.
It makes me sad
I write some of this in a jocular tone, only because that’s one way for me to deal with it. I laugh because that’s better than crying. In all seriousness, I think that experimental science can improve our lives, and it frustrates me when bad science takes up the space that could be occupied by good science.
And Steven Levitt . . . he’s got the training and experience to evaluate scientific claims! He could read Langer’s papers, he could download and reanalyze what data are available, he could google search for replications and criticisms, he could read what Mark Liberman and others had to say, etc. He doesn’t have to do that work, but, if he’s gonna ask, “How do you know whether to believe surprising results?”, then he should. Otherwise, why bother? Langer’s been interviewed a million times already; what’s the point of one more puff piece? I just don’t get it. Levitt can play a useful role in the conversation here, and he chooses not to. Really frustrated.
Again, I think it’s just fine that Langer and her colleagues do their research—not that they need my approval or endorsement! Speculative studies are part of the research ecosystem. My problem is not with high-risk, potential high-return research; my problem is the misrepresentation of scientific evidence.
They’re grifters — like Malcolm Gladwell and Nate Silver — giving credulous audience members what they want to hear: that the world isn’t as complicated as they think and that there are simple ways to control and understand it, ways that ordinary science doesn’t provide. You’re expecting them to act like scholars when they’re actually carnival barkers.
Grouping Nate Silver with Malcolm Gladwell doesn’t make any sense to me. Silver constantly harks on uncertainty and is skeptical of any individual poll or paper or explanation. If your takeaway from his presidential election forecast is “US electoral politics isn’t that complicated”, you’re not reading anything he writes.
What you wrote is certainly true of Gladwell and Levitt, however.
Yeah, incredible to link Silver here. Reasoning under uncertainty is, like, his thing.
I think Silver is one of the “public intellectuals” (not sure how precisely to categorize them, but let’s go with that) who’s most likely to be aware of base rates for extreme claims. Good chance he’s aware of Andrew’s blog and 100% chance he’s aware of the general themes/takeaways of the replication crisis.
When it comes to quantiative predictions, Silver is pretty good. However, he recently wrote a book, which, in its simplistic characterizations of people, is very Gladwell-esque.
I see the consumers have arrived. Silver sells a quantitative analysis of the Presidential election so that good liberals can feel like the world isn’t actually that complicated. We can figure out the percentages and be happy in our comfortable living room! It’s not crazy, it’s 72%! And he does it carefully enough that he can always scold critics later on…but he’s gone from major media to random website because grifters are recognizable at a certain point.
Total:
I have no idea what you’re talking about when you say “good liberals.” Lots of people of all political persuasions are interested in the election outcome. I don’t agree with everything Nate writes, but I think it’s a bit much to call him a “grifter”: he makes money selling a service (quantitative analysis and punditry) that people are willing to pay for, which seems fair enough.
For some general background, see my post, Why are we making probabilistic election forecasts? (and why don’t we put so much effort into them?).
> Silver sells a quantitative analysis of the Presidential election so that good liberals can feel like the world isn’t actually that complicated.
Again, no one who has read a single article Nate has put out in the past 15 years comes away thinking “wow, electoral polling isn’t that complicated”.
> We can figure out the percentages and be happy in our comfortable living room! It’s not crazy, it’s 72%!
We “good liberals” are definitely not “happy and comfortable” when we read Silver’s current forecast of a 55% chance of a Trump victory in the upcoming election.
> And he does it carefully enough that he can always scold critics later on…but he’s gone from major media to random website because grifters are recognizable at a certain point.
This is kind of take that I would only expect the probabilistically innumerate on Twitter or reddit to state, not in the comments of this blog. His scolding of HuffPo’s 2016 forecast, which assumed there was no covariance between different states’ polling averages, was of course sound and principled. His critique of Rachel Bitecofer’s 2020 forecast (that she was overfitting based on the 2018 House blue wave results) was less precisely stated, but still consequentially on the mark–her forecast of TX/IA/OH being toss-ups seems absurd given the 2020 actuals.
> but he’s gone from major media to random website because grifters are recognizable at a certain point.
Calling Substack a “random website” is simply an admission that one is ignorant of the modern internet.
Jack –
I think part of the problem here is that Silver’s work explicitly on polling is pretty distinct from a lot of his other political pundrity stuff and how culture wars stuff (some of which is on his substack). I greatly respect his circumscribed probabilistic stuff on polling. His other stuff, not so much. And I’m not sure that “grifter” for the other stuff is that far off the mark. He is definitely marketing a contrarian provocatuer brand.
I see the kool-aid is flavorful this week.
Oh, look, Nate, like the good grifter he is, is declaring victory.
Total:
Hey, that’s not fair. I have my differences with Nate, and, yeah, I’m annoyed that he’s stopped responding to my emails (even when I gave such a nice review to his book a few months ago!), but I don’t think it makes sense to call him a grifter. He offers a unique blend of analysis and punditry, with unique insights, good writing, and a strong persona, and people are willing to pay for that. I wouldn’t call that “grifting.”
I don’t do Twitter at all, but my impression is that his persona on Twitter is what people are responding to when they say “grifter”. I honestly don’t know, but everyone who says “grifter” about him seems to be referring to the kind of stuff he tweets.
Quite a few people on the left really dislike Silver for reasons that are unclear to me. Perhaps because they don’t like “horse race” coverage of elections. In their view all reporting on elections should be making a case for the candidates they prefer.
James:
I think Nate does good work, both as an analyst and as a pundit. He’s got his flaws, and I wish he’d be a bit more open to alternative perspectives, but on balance I’m a fan. That said, I can see how he could get on some people’s nerves. Still, I can’t see the “grifter” label fitting him at all.
I get the impression that there are lots of grifters in politics, for example campaign consultants who take candidates’ or donors’ money, promising to spend them on campaigning, but actually sending the money through a series of channels so that they end up taking much of it for themselves. Or political activists who might start out in all sincerity but end up just making money selling their mailing lists. Or whatever. But that’s not what Nate does. It should be possible for people to disagree with Nate, even to dislike him, without calling him a “grifter,” that’s all.
James –
Quite a few people on the left really dislike Silver for reasons that are unclear to me. Perhaps because they don’t like “horse race” coverage of elections.
I’ll give my take on this, with some detail.
I respect Nate’s work on polling a lot – including what he writes on the topic of how uncertainty plays out in the polling and other peripheral aspects of looking at the polling.
However….Nate does quite a bit of what I’d call hot takes on issues like the lab leak, which I think are not grounded in a careful analysis of the evidence, but are more based on fairly facile reasoning in line with his ideological agendas. That’s ok, there’s a lot of that going around these days (myself excluded, of course), but I think Nate does it in a way to attract attention by being provocative and ruffling the feathers of “the woke.”
As another example, he did quite a bit of speculation about why Harris should have chosen Shapiro instead of Walz for the VP pick. Some of that was interesting – where he took a historic look of what polling data say about the potential impact of a VP pick on the ticked. Some of it, I thought, was quite lame – as it only looked at the downsides of the Walz pick and the upsides of the Shapiro pick in support of his view that Shapiro would have been the better pick. Then, multiple times he referenced that lame speculation in a kind of repeated “I told you so, they should have listened to me.”
Sure, i’m doing a lot there that could be considered as a kind of biased speculation, but yeah, I think he’s pretty arrogant and self-impressed. Of course I could be wrong about that – but I think my reactions are fairly common. When people refer to him as a “grifter,” I think often they’re talking about what comes across as a kind of arrogant and juvenile provocativeness, and in the social media environment that is often seen as trying to attract “clicks and likes” – which in turn attracts revenue (e.g,, substack subscriptions, book sales, etc.).
Hey, it’s his right and he’s clearly good at it. He attracts a lot of clicks and likes and it’s of course it’s his prerogative. But it’s also my prerogative to judge his public profile.
One last piece. Nate clearly indicates that he seems to think a lot of the pushback he gets is a manifestation of his being a truth-teller to people who are resistant to his keen insights. For example, that could play out as a view that established virologists and immunologists are just pushing back through gatekeeping against “lab leak” theories so as to protect their petty kingdom. Hey, could be. But a basic underlying factor there is that despite Nate’s healthy view of his own intellect and ability to evaluate probabilities – it’s just a fact that he lacks the domain-relevant knowledge to assess such a complicated and technical topic. That kind of supreme confidence among “smart” people to weigh-in on topics where they lack background expertise is, I think, rather epidemic these days. I get that the value of “experts” deserves skepticism – but so does the lack of respect displayed by people who narrative surf and earn a lot of bucks for offering their opinions. As a question of uncertainty, I think Nate takes a pretty sloppy approach when he weighs in to offer a probability assessment on whether COVID involved a lab-mediated versus market-mediated spillover – when it’s largely out of line with the views of those who have the most relevant background knowledge.
Joshua:
What you’re saying is that Nate’s not just an analyst, he’s also a pundit. Analysis and punditry are different skills! I think we should compare Nate’s punditry to that of other pundits. Measured in this way, I think Nate’s a strong pundit. He’s made some mistakes as a pundit (in 2015, giving Trump only a 2% chance of gaining the Republican nomination, even when Trump was leading in the polls; in 2021, labeling Eric Adams as the possible future of the Democratic party), but pundits make mistakes: that’s an inevitable consequence of issuing a series of hot takes. Taken as a whole, I think Nate’s a good pundit. I think Nate’s lab-leak punditry is fine too. Maybe it’s not as solid as his analysis, but, again, I think the right comparison is Nate-as-pundit to pundit-world, not comparing Nate-as-pundit to a hypothetical ideal Nate analysis.
Maybe it would be better for Nate to just do analysis and not punditry, and maybe it would be better for me to just do statistics research and not write blog posts—but we do what we enjoy.
Andrew –
It strikes me that maybe I’m holding Nate to an unfair standard. I first knew of him as a top rate polling analyst who wrote very educational essays about polling and the related treatment of uncertainty. I learned a lot of new stuff. So I built up a high opinion. Then I see him going on twitter pounding out what I consider very run-of-the-mill punditry. And I see him giving pretty empty takes on a complex topic like COVID origins – where it’s readily apparent that he hasn’t actually investigated the details in-depth but just goes with off-the-shelf heuristics about scientists protecting their turf. So I’m disappointed because what’s run-of-the-mill punditry doesn’t meet the high standards I’ve built up.
I suppose if I had only been exposed to his punditry, I’d just go “Meh, another “smart” guy out there making a lot of money off of poorly researched hot takes” and not think much of it. But because of the high standard he built up in my eyes, his “meh” punditry somehow feels worse than run-of-the-mill even if it’s not necessarily any worse than the next hot take guy.
“…But that’s not what Nate does. …”
Well in 2016 and/or 2020 he was accused of deliberately inflating Trump’s chances to make the election appear more competitive than it was and drive traffic to his web site. I think this would qualify as grifting if it were true. But it seems like a case of shooting the messenger. People wanted to believe Trump had no chance and got mad at people who said otherwise. And if anything got even madder when those people proved to be correct.
And…
and maybe it would be better for me to just do statistics research and not write blog posts—but we do what we enjoy.
Your blog posts don’t, at all, seem to me of a different grade of quality than your more technical stuff (that part of it I can understand). I think Nate’s punditry is of much lower quality than his analysis. So I don’t think your comparison applies. So sure, I don’t begrudge him doing what he enjoys, but that doesn’t mean I have to leave behind my right to critique.
Are the teachings of psychology “beliefs of mainstream science”?
Reading Paul Meehl, who was head of the APA, it was definitely not already in the 1960s.
It seems to have started out as science (studying reaction times to improve astronomical observations), and behaviorism largely continued this legacy into the 1940/50s, but that is a very small subset of what gets called “psychology”.
Using the term research instead, are these even mainstream research-based beliefs?
I don’t think mainstream researchers in other fields (eg, physics) regularly accept those beliefs.
My point is its more like “surprising results that challenge the beliefs of wild speculators”. They established their beliefs on a solid foundation to begin with.
typo: *never* established
Btw, itd be great to get some kind of preview confirmation, especially on mobile.
Andrew, I think that you miss the point of “Freakonomics”: it’s to entertain (cf. Malcolm Gladwell), not to do anything intellectually deep.
Victor:
I think you’re letting them off too easy here. They’re portraying their claims as supported by good evidence, not as pure speculation. A key part of the whole Freakonomics brand is that they are hard-headed and learn from evidence.
Certainly the point is to entertain, but implicitly it is not pure entertainment – it is entertainingly informing you of interesting and seemingly true things. The idea that it is more than simply entertainment should be clear if you imagine they said ‘none of the scientific research we discuss is real’ or ‘none of the claims we make are well-founded’.
This reminds me of Merton’s norms, specifically “organized skepticism”. To me, Levitt is exhibiting *dis*organized skepticism: he is skeptical of mainstream science, but then credulous of this one researcher’s work.
Come to think of it, a lot of the other names that pop up on this blog violate the norms. For example, Wansink (and many others) violate the communism/communality norm, by not sharing data. Disinterestedness is violated every time there is a conflict of interest. Universalism is violated every time claims are evaluated based on pedigree of the claimants (I suppose this applies to the professors affronted when some upppity independent researcher dares to question their research).
Adede:
It’s complicated! I don’t think sharing data is the norm in science. Maybe it is becoming the norm, but in the time when Wansink was active, it was not the norm. Universalism is an interesting one too, in that the standard practice is to make universal claims in the title and abstract of a paper and then retreat to particularity when replications fail.
In general freakinomics has been and still is a positive influence in today’s world. Yeah every now and then they mess up, I don’t agree with everything they say, but I can say that about 100% of the people I hear and interact with.
That said they can accept Andrew’s criticism and get better. But I wouldn’t go so far as to call them grifters. There are way better and nefarious examples of grifters in our society today. We can be more Bayesian in our deductions here imo.
Whether or not you agree with the term grifters, I think the larger point that their primary interest is not “rigorous scholarship in search of the truth” but rather entertainment/audience engagement still stands. That’s where the money is. These types of non-scholarly incentives on the part of researchers are often discussed on this blog, whether conscious or sub-conscious, so I am a bit surprised at the surprise here.
Anon:
I’m not at all surprised at what Freakonomics is doing here, any more than I’m surprised at what that celebrity physicist Carroll was doing on his podcast, any more than I’m surprised that Harvard professor Langer is pushing this stuff, any more than I was surprised that Columbia Prof. Oz was promoting dubious health supplements.
I’m not surprised. I’m annoyed, both at the people who do this sort of promotion and the institutions (Harvard, Columbia, NPR, Ted, etc.) that sustain them.
And I’m also saddened, because . . . Levitt etc. already have earned tons of money and tons of fame. To promote junk science just to get a little more money and a little more fame, or just to stay in the good graces of various science celebrities . . . that’s just a pitiful goal to have.
Thanks for the clarification, I wholeheartedly agree.
Anonymous –
Whether or not you agree with the term grifters, I think the larger point that their primary interest is not “rigorous scholarship in search of the truth” but rather entertainment/audience engagement still stands. That’s where the money is. These types of non-scholarly incentives on the part of researchers are often discussed on this blog, whether conscious or sub-conscious, so I am a bit surprised at the surprise here.
I think that’s unfair. It’s not an either/or. Those goals aren’t mutually exclusive and you can’t reverse engineer from individual (or even sometimes repeated) instances of poor standards to sxclude an interest in rigorous scholarship. And certainly if we’re going to use “pursuit of self-interest” heuristic to exclude an interest in rigorous scholarship then we could argue there are no examples of rigorous scholarship anywhere.
To me, it seems the extent of their positive influence is being publicly corrected by better statisticians and indirectly raising awareness of better practices. Or, lacing OTC meds with arsenic is good because we’d get better anti-tamper mechanisms afterward.
Andrew -.
. It is problematic to take survey responses as measures of actual behavior, especially in the context of a study of an intervention specifically designed to alter perceptions of exercise.
If they didn’t use any objective measures of physical activity and/or diet didn’t hey at least mention that as a caveat in the limitations section?
If they didn’t, then there’s no excuse for taking their results at a sort of face value.
If they did, then there’s no excuse for taking their results at a sort of face value.
I can’t imagine where these days, a researcher would not mention the inherent limitations of survey data on diet and physical activity in a limitations section, let alone publish results based on measuring physical activity and diet without using objective measures. Of course doing so makes a study more complicated…but c’mon.
“…let alone publish results based on measuring physical activity and diet without using objective measures.”
Well, physical activity and diet, per se, are not fields I follow closely and have direct interest in. But I do a fair amount of research in patient self-management of diabetes, which necessarily draws on those literatures. From what I have seen, the use of objective measures of physical activity and diet in these studies is uncommon. (If somebody who knows these literatures better than I do wants to challenge me on this, I’d be happy to stand corrected.)
To really get reliable, valid data about either of these requires highly intrusive surveillance of study subjects. Typically this is impossible with people undertaking their normal daily activities, and, when done, tends to be carried out in special environments with participants being temporarily resident there for a brief period. Of course, this, in turn, limits the applicability of any findings to real-life conditions.
It is also incredibly expensive to do that kind of research. So I don’t find it surprising that most studies rely instead on self-reports. Sometimes they will try to structure the self-reports a bit, e.g. asking people to keep food or exercise diaries, but I doubt this improves the measurements by very much. I recall vividly one study I was involved in where we were trying to measure fast food consumption and asked people to save and turn in to us their itemized receipts. But participant acquiescence to those directions was abysmally low. (And even had it been 100%, we still would have had to rely on participant self-report to distinguish which items they actually consumed–sometimes the receipt included items purchased for other people who were not in the study.) The difficulties are just endless.
It has long been a fantasy of mine that someday, somebody would invent a tiny sensor that could bonded to the surface of a tooth and that would accurately record all food intake and transmit it to a database. But as far as I know, nobody has attempted to do that. And I have no idea if it is even feasible.
Clyde –
I’m trying to limit my comments but wanted to make a quick response.
I happen to have seen a fair amount of research that uses things like biomarkers, cell phone apps (for taking photos of what people eat), food diaries (I’d guess somewhat better than recall surveys?), double-labeled water, accelerometers, Dexa scans, pedometers, etc. Of course none of those are perfect, and it’s enitely possible that what I’ve seen isn’t representative. But I think any research these days that relies heavily on recall surveys should necessarily be clearly caveated, and taken with a heavy grain of (carefully measured) salt. I would imagine with wearable devices, things should be getting considerably better soon.
Langer’s troubles go back a long time, although this one is conceptual, not about data:
https://psycnet.apa.org/fulltext/1980-03618-001.pdf
(Langer replied by questioning the relevance of base rates.)
“I think that experimental science can improve our lives, and it frustrates me when bad science takes up the space that could be occupied by good science.”
Me too. I’m tempted to do something about this post-graduation.
For news more broadly, there are alternatives like the Positive News Foundation (https://positivenewsfoundation.org) and Reasons to Be Cheerful (https://reasonstobecheerful.world/category/education/). The analogy here would be to create a space online for rigorous science to be vetted and discussed to give laypeople/practitioners something solid to circle around.
If anyone knows of any precedents (the anti-Freakonomics and the anti-Mindscape media outlets), drop a line here.
The phrase “high-risk, potential high-return research” caught me by surprise in the context of the research being discussed in this blog post. I suppose there are some reputational consequences for producing ‘counterintuitive’ research, but the bar for what counts as ‘counterintuitive’ in psych. seems ‘esp.’ (pun intended) high. This kind of research just seems par for the course and that’s the real issue. In political science, a lot of the ‘risk’ seems just like ideological risk.
Gaurav:
Studying extra-sensory perception, for example, is high-risk, high-return in the sense that there is a high risk that your experiment will be a failure, but if you really find something, it’s a big deal. I associate the phrase “high-risk, high-return” with funding pitches; in this case, the risk is the funder’s risk that it is wasting its money.
Andrew is correct and Steve Levitt and Ellen Langer should feel ashamed. This is what Langer claims, and Levitt enthuses over:
That is EXACTLY the world view of my former boyfriend at Stanford, Brian Wansink (Stanford Graduate School of Business, PhD in marketing for him; engineering for me). As Andrew has said before, overhyping mindfulness can morph into mindlessness.
Remember that Langer’s research subjects are men in their 80s. By pretending they were living in 1960, Langer claims that these 80 to 90 year old men experienced measurable improvements in hearing, memory, strength, vision, and appearance?!!! These are the sort of claims that charlatans make. It isn’t merely contrarian or counter-intuitive.