Jonathan Falk came across this article and writes:
Is there any possible weaker conclusion than “providing caloric information may help some adults with food decisions”?
Is there any possible dataset which would contradict that conclusion?
On one hand, gotta give the authors credit for not hyping or overclaiming. On the other hand, yeah, the statement, “providing caloric information may help some adults with food decisions,” is so weak as to be essentially empty. I wonder whether part of the problem here is the convention that the abstract is supposed to conclude with some general statement, something more than just, “That’s what we found in our data.”
Still and all, this doesn’t reach the level of the classic “Participants reported being hungrier when they walked into the café (mean = 7.38, SD = 2.20) than when they walked out [mean = 1.53, SD = 2.70, F(1, 75) = 107.68, P < 0.001]."
Across many fields, it’s hard to escape the conclusion that the aim of most research is to keep researchers off the streets.
I like Andrew’s negative riffs, but I also like to steelman things at the same time as reading those critiques. Even if this execution is not so great, I think high level idea is salvageable. So say someone did an experiment with a restaurant; Menu A with calorie and Menu B without, and then measured overall calories consumed – not so bad an idea right? Similar here to the healthy index (ignoring issues with confounding/selection, but similar idea).
Knowing the size of the effect I think could be useful. So say someone wants to make all restaurants display this info, it takes time and money to do that. I do the experiment, and show the effect size is a marginal reduction of something like 100 calories. I don’t know if 100 calories is big enough to warrant making restaurants go through the effort, but at least it gives something to debate. If effect is very small, then easier to say not worth it. (Or can have $$$ incentives for restaurants to do this.)
No argument about execution though from me for this paper to be clear. I review pysch research papers on occasion, a not fun time is to try to replicate the different in text Chi-squares a F stats.
I will admit I do like the idea of nudges for many situations, I will try to keep my inner over-hyped nudgelord in check though.
“I think high level idea is salvageable.” I don’t dispute that the high level idea is useful and even important; I doubt anyone disputes this. That doesn’t mean any study that brushes up against the high-level idea is inherently worthwhile.
Andy, yeah, I don’t understand the anti-nudge hate. Some nudges work! Honestly it’s hard for me to believe people dispute this. Sure, mock the bad nudge research — mock all bad research! — but I feel like this blog tries to give the whole idea of the ‘nudge’ a bad name that it doesn’t deserve.
Phil:
If the “nudge” industry just said, “Some nudges work!”, then, sure, I’d have no problem. But they make much stronger claims than that. See for example this ridiculous meta-analysis (and followup here).
The other annoying thing about the nudgelords is that they can’t seem to learn from their mistakes, as discussed here and here.
But, yeah, there are good things in the nudge movement. That’s one reason it upsets me so much! If it were 100% crap, like those features on TV about the Bermuda Triangle or ancient astronauts or the search for Noah’s Ark or whatever, I’d just laugh. When it’s pernicious conspiracy theories like the election deniers or the anti-vaxx movement, I get angry because these people are actively making the world worse. With nudge, my annoyance is different: they’re taking some interesting science and doing a lot of research, but they choose to promote it through hype, they routinely overstate their findings, and they seem to be working overtime to avoid thinking about what could possibly be doing wrong with their research. They’re living in the happy-talk Ted/NPR celebrity bubble. And it makes me mad. Not as mad as I get about the conspiracy pushers, and it’s not as stupid as the stuff out there on UFOs, ghosts, ESP, etc., just super-annoying in its own way, that these people are so well connected in academia, business, and government, and they use those connections to promote bad science.
I guess you pay more attention to Big Nudge than I do.
I still think you’re throwing out the baby with the bathwater. Hate the nudgelords, not the nudges!
The conclusion’s first statement of association is fine – correlation, not causation. But the second statement suggests causation. I don’t see the basis for suggesting that providing caloric information may help some adults with food decisions. It is equally plausible that those adults who use caloric information also have healthier eating habits – pure correlation with no causation to infer. I’m not purchasing the paper to see if they have any reason to suggest causation, but it seems unlikely from the information in the link. So, I’m not sure that the statement “is so weak as to be essentially empty.” I see it as actually stronger than that – but not substantiated by the evidence.
I don’t understand what you mean. The opposite of “providing caloric information may help some adults with food decisions” would be “providing caloric information won’t help any adults with food decisions,” which is false even if one adult anywhere makes a decision using calorie information. We all know, therefore, that the quoted statement is true; we don’t need the study or the data to tell us this. (“We all” unless we’re the type of skeptic who thinks nothing is knowable, but there’s nothing to be gained by such views!)
Perhaps you mean that *this study* didn’t show causality among its participants, but you can’t assess whether it did or not from reading the abstract. That’s what the paper is for!
I wasn’t focused on the “all” aspect. Certainly, any paper that claims that “some” people may be helped is trivially correct. What I was looking at was the association between reading caloric labels and improved diet. While I believe these are associated, I don’t see how it “suggests” that providing such information may lead to improved diet. It just as well may not lead to improved diet, if the study is merely observing that those people who read labels are more careful about what they eat.
Note that the study did not perform an experiment where some people got such information and some did not. It was an observational study where people self-report whether or not they use such information.
Dale –
> . I don’t see the basis for suggesting that providing caloric information may help some adults with food decisions. It is equally plausible that those adults who use caloric information also have healthier eating habits – pure correlation with no causation to infer.
Sure. But maybe some individuals with healthy eating habits use caloric information to make healthier choices. It’s not only that eating habits is a confounder. IOW, eating habits plays a moderating role in the causal relationship between caloric information and food decisions.
No?
1. Andrew is quoting a paper; he didn’t write that sentence.
2. I can’t say why the original authors reported an F-test, but t-tests and F-tests for comparing means like this are not different tests: a t-test is just a special case for comparing two groups, while F-tests apply to any number of groups and are thus more general. F=t^2, so in this case t= 10.4. Many people have learned that a t value a bit more than 2 for even a modest number of degrees of freedom (~20) is conventionally significant. There’s no similarly simple rule of thumb for assessing F-values (because the distribution of F depends on both the among and within group degrees of freedom).
Actually, no. When teaching statistics, I teach the general linear model, with regression, anova, ancova, etc. as particular applications. I cover t-tests only as a special case of an anova with 2 groups, which is better done as an F-test with df-among=1. I include t-tests because they are so commonly encountered in the literature.
I also include t-tests because I like to include elements of the history of any subject I’m teaching, and you gotta like a test developed at the Guinness Brewery in Dublin!
In the US, when you see 100 calories from fat on a label, that means less than 120 calories. On the other hand, 100 calories from carbs means *at least* 80 calories. IIRC, then vitamins are labeled according to “at least” method but cholesterol is “less than”, and so on. So the interpretation depends on what you are looking at.
So I doubt people really understand these labels. Besides the confusing meaning, how accurate are the labels anyway? Eg, that data should be associated with a date, location, and methodology somewhere.
If manufacturing, etc changes then it should be retested. But is it really healthy to be eating anything where its feasible to regularly test like that? Probably best to eat stuff that has no label at all.
And how did they determine a “healthy” diet?
You can’t just skip this stuff and compare two averages then expect to predict the future or perform feats. For publishing an endless series of conflicting papers, it is perfect though.
I’m not seeing the problem here. In courses, I’ve found it hard to extract any systematic ideas from methods instructors. How, then, should I move from Evidence -> Claim? What is a reasonable way to do so? Can you suggest a rewrite of the above claim the way you did with academic papers’ titles a few months ago?
In practice, I don’t mind an overly hedged claim because readers will naturally hype up claims a bit.
It *is* hyped up. Did they check the calorie labels actually reflect what was in the meal, or even where the info came from? Also how can you use this to assess “healthy diet quality” when the healthiest food (fresh fruit/meat/vegetables from an area of fertile and uncontaminated soil) has no labels on it at all?
They should teach to find out where numbers on the page come from before you can draw a conclusion. Alternatively, you need to explicitly mention the heuristics being used so people can trace them back to the original source.
That seems sensible (just now skimmed the original paper). I would be curious what worked examples of that kind of heuristic tracing would look like concretely. But it reminds me of encounters when PIs have informed me to write my Limitations sections in less rigorous/honest ways. Typically, we don’t report all true limitations (just the ones we’ll get away with like lying about your greatest weaknesses during a job interview).
Just add a line in the conclusion/abstract like: “Assuming our caloric labels and definition of healthy diet are valid …”
The details of those assumptions should be expanded on elsewhere in the paper, citing the original sources and probably some sources that investigated their validity.
Personally, I advocate writing down *all* the assumptions you can identify then trying to elicit more that you missed from anyone who (proof-)reads your paper.
Basically it is that if you are willing to accept premises x + y + z + …, then according to logical argument L we can deduce C. Make the premises and structure of the argument explicit.
And even then there can be another set of premises x’ + y’ +… that also explain the observations.
Jay: the claim has no substance. If the claim has no substance, then it’s reasonable to infer from the claim alone that the study does not provide evidence of anything.
The only sensible claim to make is that the study design was insufficient to provide evidence regarding the benefits of caloric labels on menus.
https://en.wikipedia.org/wiki/Experimentum_crucis
I use calorie information to avoid ordering low calorie meals. I wonder how many people are like me.
I think the original statement quoted is phrased specifically to not exclude this possibility, which I think was raised as a possible consequence of labelling legislation. I believe I have seen it in action, though I was not the decision-maker. I was eating with friends at a cafe and watched as one of them decided which dessert to purchase to share around (I don’t eat anything with added sugar, so I was just an observer). They paid careful attention to the calorie figures for the various cakes and shortbreads on offer, and selected the highest calorie treat available, as the best value for money. I asked them afterwards to confirm that this was what they were doing, because I was interested to see this predicted behaviour “in the wild” so to speak. Their track record is such that I believe that they were perfectly sincere, not putting on any sort of show for me or others to make a political point.
I like this blog, but not when people don’t read the actual studies. That line about “providing caloric information may help some adults with food decisions” was just a bit of outro fluff. The study reported some interesting findings – e.g., no differences between BMI groups in calorie label use or in benefits of use, which translates into one more argument against fat shaming. (The BMI may be flawed, but the data at least show that high-BMI folks aren’t more likely to ignore calorie labels or adjust their food choices when reading them.)
Ken:
The above post did not criticize the substance of the paper. What I wrote is, “the statement, ‘providing caloric information may help some adults with food decisions,’ is so weak as to be essentially empty,” which I think is consistent with your description of that sentence as “outro fluff.” The rest of the paper could have great stuff, for all I know.
Andrew –
Above you say this:
> I wonder whether part of the problem here is the convention that the abstract is supposed to conclude with some general statement, something more than just, “That’s what we found in our data.”
Hmmmm. I think that’s the point. Just saying “that’s what we found..” seems insufficient. Kinda like saying “after spending a lot of time and money and effort, we have concluded that the data aren’t adequate to assess whether masks work or not.”
Except when people DO say that, as in the Cochrane Review, people misinterpret or mislead what they said.
In that context I think that saying something essentially empty isn’t really that bad. It’s not as clear as saying something that’s explicitly empty. But it could be (and often is) a lot worse.
Joshua:
Agreed: a vacuously true statement is preferable to a false statement.
To that end, I recommend that all abstracts of published papers end with the statement, “1 + 1 = 2,” so that the truth of the statement will never be in doubt.
to defend the study, isn’t the point that it is necessary for them to notice calorie labels and then use them? So, for this intervention to work/help people, a chain of non-trivial things have to occur? But once that chain of things occur, there is a noticeable improvement?
I still don’t see it that way. The chain – notice labels – read them – change behavior cannot be inferred from this survey data. All we know is that the people who self report using the labels have healthier diets. The chain can just as well be eat healthy – read labels with no change in behavior due to the labels. And, the main focus of the study – that the effect does not seem to vary with BMI (aside from all the concerns about that measure) “suggests” to me that the labels are not causing any change in behavior. I would have expected to see health status moderating the impact of the labels if they were really effective.
Still, it is easy to say that providing labels is a good idea and won’t do any harm. But I would propose the following 2 hypothetical studies to think about. Study 1: we find that people who read their detailed credit card agreements (those 20+ page small print documents) practice better financial decision making. Therefore, it suggests that requiring such disclosures is a good thing. I can go along with that as well – after all, more information is good, right? There is still an issue of what information to provide and the possibility that too much information will adversely affect decision making. But I’m willing to conclude that such information should be provided (and required).
Study 2: We find that universities that require applicants to provide diversity, equity, and inclusion (DEI) documents have a more diverse faculty. Therefore, we require all universities to include DEI as part of the application process.
Now, I realize study 2 is fraught with additional emotional-laden concerns, and policies “requiring” anything involve more than just a statistical finding. But I would propose that the same potential confusion between correlation and causation would be at work with these hypothetical studies. The danger is misreading the correlation as causative is that the conclusion (to require provision) diverts attention from what the real causes are (whatever these may be). DEI statements may not do anything to improve outcomes, notwithstanding the apparent (hypothetical) correlation that is found. Similarly, caloric labeling may not improve eating behavior, but we are given the false impression that we have addressed a real cause of unhealthy eating.
I still think the study’s “suggestion” is unwarranted.
Without commenting on the specifics of this article/abstract, I would like to point out that “water is wet” studies are actually useful. Even if the results are obvious, it’s good to confirm what “everyone knows” with some actual data. Besides, who exactly is “everyone” and is what they “know” actually true?