Ethan Ludwin-Peery writes:
I was reading this paper today, Ultra-Processed Diets Cause Excess Calorie Intake and Weight Gain (here, PDF attached), and the numbers they reported immediately struck me as very suspicious.
I went over it with a collaborator, and we noticed a number of things that we found concerning. In the weight gain group, people gained 0.9 ± 0.3 kg (p = 0.009), and in the weight loss group, people lost 0.9 ± 0.3 kg (p = 0.007). These numbers are identical, which is especially suspicious since the sample size is only 20, which is small enough that we should really expect more noise. What are the chances that there would be identical average weight loss in the two conditions and identical variance? We also think that 0.3 kg is a suspiciously low standard error for weight fluctuation.
They also report that weight changes were highly correlated with energy intake (r = 0.8, p < 0.0001). This correlation coefficient seems suspiciously high to us. For comparison, the BMI of identical twins is correlated at about r = 0.8, and about r = 0.9 for height. Their data is publicly available here, so we took a look and found more to be concerned about. They report participant weight to two decimal places in kilograms for every participant on every day. Kilograms to two decimal places should be pretty sensitive (an ounce of water is about 0.02 kg), but we noticed that there were many cases where the exact same weight appeared for a participant two or even three times in a row. For example participant 21 was listed as having a weight of exactly 59.32 kg on days 12, 13, and 14, participant 13 was listed as having a weight of exactly 96.43 kg on days 10, 11, and 12, and participant 6 was listed as having a weight of exactly 49.54 kg on days 23, 24, and 25.
In fact this last case is particularly egregious, as 49.54 kg is exactly one kilogram less, to two decimal places, than the baseline for this participant’s weight when they started, 50.54 kg. Participant 6 only ever seems to lose or gain weight in increments of 0.10 kilograms. Similar patterns can also be seen in the data of other participants.
We haven’t looked any deeper yet because we think this is already cause for serious concern. It looks a lot like heavily altered or even fabricated data, and we suspect that as we look closer, we will find more red flags. Normally we wouldn’t bother but given that this is from the NIH, it seemed like it was worth looking into.
What do you think? Does this look equally suspicious to you?
He and his sister Sarah followed up with a post, also there are posts by Nick Brown (“Some apparent problems in a high-profile study of ultra-processed vs unprocessed diets”) and Ivan Oransky (“NIH researcher responds as sleuths scrutinize high-profile study of ultra-processed foods and weight gain”).
I don’t really have anything to add on this one. Statistics is hard, data analysis is hard, and when research is done on an important topic, it’s good to have outsiders look at it carefully. So good all around, whatever happens with this particular story.
I would suspect the scales have a “memory” function, and attempts not to report widely divergent results (which triggers customer complaints). From the methods:
That scale does store previous results, but I couldn’t tell from looking it up exactly how it works. Perhaps not the scale itself, but looks like they smoothed the data. While 1 kg is not a meaningful amount of overall weight gain/loss over two weeks, it definitely is if somehow water weight was accounted for. I don’t really see how this linear regression does that though.
I’m gonna be pissed if this was a case of data fabrication because this was absolutely the best diet and health related study ever done if it was done as described. They’ve got people in a calorimeter and everything.
I wonder if the linear regression is somehow from measuring them multiple times and then running a line through the trend for the last n days, kinda like a loess. Then if there was a 0 trend you’d see the same exact numbers for several days in a row, at least when rounded to 2 digits
User manual says:
https://mfimedical.com/products/welch-allyn-scale-tronix-5702-mobile-bariatric-stand-on-scale
If that is accurate I’d guess the scale wouldn’t display values like 96.43, it would round to nearest 96.45 kg. So then those values are not the readouts from the scale. Really it is hard to say for sure without playing with one of these scales.
Would be silly to use a scale which does not report raw data, for such an elaborate experiment, no?
Lots of silly things happen in research. I’d even say the sillier and more absurd it is, the more likely it is to slip through the cracks. Especially when it comes to averaging.
Eg, see the further comments below about pajamas and sig figs. Then averaging pre- and post-meal hunger together. Also, they didn’t measure the main outcome (energy intake). The nutritional info for the meals was estimated from a database of ingredients that is probably filled with guesstimates (made years ago) itself.
But overall this is one of the better nutrition studies Ive seen.
There’s really no such thing as a scale that reports “raw data”. The raw data is a voltage across a resistor in a bridge network. That voltage fluctuates with the length of some metallic traces which stretch or contract with the deformation of a piece of metal based on the weight applied to the metal through the plate on which the subject stands, the voltage is then used together with an assumption about the gravitational acceleration at the location to be converted through some formula to a measurement in kg. The force applied by the subject varies based on how much they shift around or whatnot. The only way to get a meaningful number out of it is to low pass filter the voktage, or sample the voltage at high frequency and then digitally filter the sampled data. You’re always going to have some stabilization technique otherwise the least significant figures would be unusable. I agree they shouldn’t be doing historical averaging etc but it wouldn’t surprise me at all if the scale reports more significant figures than is really available. For example it might read to nearest 0.1 kg but really only measure to the nearest 0.5 kg or 0.2kg or something.
Reading the linked retraction watch article it seems clear that some of the trailing decimal issues are due to changing the supplied pajamas
https://retractionwatch.com/2021/01/21/sleuths-scrutinize-high-profile-study-of-ultra-processed-foods-and-weight-gain/
I don’t think there’s going to be any fabrication here.
However the pictures of the data collection form make it clear the scale weighs to the nearest 0.1kg and then pajamas were weighed on a more sensitive scale to nearest 0.001kg they subtracted the pajama value from the human value to get the weight without pajamas. This is not a good scheme as the measurement error in a measurement like 85.3kg could be as high as 0.2 kg without too much difficulty. I’ve seen that kind of thing in scales where in order to avoid rapidly fluctuating values they have a low pass filter that can result in biases etc. The subtraction of the pajamas gives the impression of much more accuracy than is really justified. Ideally you’d report both the numbers in a table and then do some Bayesian measurement error model on the data.
Also can we please just shoot SAS in the head an bury it? That’s what they used for their analysis. So sad that it’s still widely used. It was a horrible dinosaur in 1998 when I first had to deal with it.
Yea, I later saw that reading one of the blog posts. I still don’t get how whatever they did to smooth the data* is supposed to account for water weight though. Eg,
https://slimemoldtimemold.com/2021/01/21/investigation-ultra-processed-diets-by-hall-et-al-2019/
Losing ~10 lbs in water weight over two weeks is on the high end but not really surprising. Really all of this is consistent with my own experience eating a meat + vegetable (essentially, lazy keto) diet. I lose weight because I feel less “hungry” (actually I would call it carb cravings), so I eat less. Also after a few days I will lose ~5 lbs of water weight that can be gained right back as soon as I eat pizza or whatever.
Why wouldn’t we expect a large correlation between calories consumed and weight change? Especially if the participants were all approximately equally active while in the study.
* And is it the “raw” or smoothed values they shared in the SAS files? I’m not willing to spend the time to convert them and look.
Also can we please just shoot SAS in the head an bury it? That’s what they used for their analysis. So sad that it’s still widely used. It was a horrible dinosaur in 1998 when I first had to deal with it.
It’s horrible now but I ran into in the late 1980s and it was a breath of fresh air after SPSS.
Anyone who confesses their sins from the 1980s is absolved. But by 1995 SAS was over it just had to keep sucking down that sweet sweet rent.
My very first real life experience attempting to apply statistics, I worked it out by hand but could not get the same answer as SAS or SPSS. Eventually I figured out there was something like an n rather than sqrt(n) in the denominator. The wasted time trying to figure this out meant it was also my last time using closed source stats software.
https://mikebyrnehfhci.wordpress.com/2015/08/03/translating-spss-to-r-mixed-repeated-measures-anova/
The bug was caused because there was an error in the original publication from the 1970s or 80s they based the calculation on. The paper was corrected a few years later but the software wasn’t until a few years ago when SPSS added an “enhanced” version that corrected the bug.
This also relates to the discussion of the scale possibly averaging historical weights. When you get odd results the first thing to check is the tools, you never know what weird bugs/malfunctions there may be.
And I also bet that SPSS devs checked their results against SAS and vice versa, thus managing to incorrectly “validate” each others algorithms. This is very similar to the failure mode we saw (and continue to see) for the pcr covid tests. When compared to each other they agree ~98% of the time, but there is little better than 50% agreement (either positive or negative) with presence of actual infectious virus under real life conditions.
One of the really great things about Bayesian modeling is you get to define the mathematical calculation to be carried out. Once a sampling scheme has been validated against a database of problems we can apply the same debugged sampling process to whatever model we like.
Of course you have to be aware of issues with models that are hard to sample, etc but at least because of the general purpose issues checking of sampling with chain plots and rank plots etc is a common process.
I agree with your first sentence but don’t follow the whole thing about sampling. The sampling scheme is part of the data generation process and thus should be specific to your model.
What I meant was drawing samples from a posterior distribution like what Stan or JAGS does, not a sampling scheme for choosing experimental units etc. Some models it’s hard to get those posterior samples for mathematical reasons.
Maybe I get what you meant now.
Rather than deriving an analytical solution for each specific (in practice: only for easy-to-solve) data-generation processes, MCMC lets you easily play around at the model level.
For sure. But I think that is more comparing numerical vs analytic modeling.
Using (wide) uniform priors is essentially the same as a frequentist model imo. Or rather, frequentist models primarily are that way because it is easiest to get an analytical solution.
But yea, I could care less about repeated measure ANOVA sphericty violations at this point. Just model the entire process you think generated the numbers you have.
Recall this blog of a few years ago
The feather, the bathroom scale, and the kangaroo
Posted on April 21, 2015 9:18 AM by Andrew
https://statmodeling.stat.columbia.edu/2015/04/21/feather-bathroom-scale-kangaroo/
“I like the weighing-a-feather-while-the-kangaroo-is-jumping analogy. It includes measurement accuracy and also the idea that there are huge biases that are larger than the size of the main effect.”
Five days after his initial blog post, Nick Brown posted an update stating that the author’s responses had been added, and concluding “I believe that these responses adequately address all of the points that I made in the original post.”
Seems like there really is nothing to see here.
The study in question is one of the rare properly designed trials in a sea of nonsense that passes for ‘nutrition science’. This was just something Kevin Hall did on the side, with surprising results. All done in highly controlled in metabolic wards, where everything is adjusted and accounted for. Weight loss is irrelevant as the bigger picture is dismantling so-called carbohydrate-insulin model.
https://www.youtube.com/watch?v=zumrAR5qHX0
Basically, the best takeaway from this particular study is that processed food group, when allowed to eat ad libidum, ingested more calories, on average. Weight loss is irrelevant, as it doesn’t equate to health.
We need more studies like this one and less nonsensical ‘interventions’ (self-reported food intake, leaflet/pamphlet distribution, etc.)
Interesting presentation, thanks. I wonder about this though:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7946062/
Why do they combine pre-meal hunger with post-meal? And then why is this adjusted for energy consumption? This seems strange.
I am having fun updating R so I can install a tidyverse package to read in this SAS data now…
I’d guess the data should be in the hungrysatiety file. R dput is here: https://pastebin.com/FHv6r7f3
They completed the surveys three days per diet, and the columns are the averages for that day. Each subject should have ~9 observations per day according to the methods. Anyway, looks like we can’t compare the pre-meal feelings of hunger from the data provided.
Also, I wonder if any of the participants opted to skip meals entirely. I suspect they felt obligated (and bored) to at least eat some of the food, which may dilute any diet effect on consumption.
That’s handy…
I didn’t download the rest of the data but right there is a contradiction with the paper. The paper says that all screened participants completed the study (20). But right here 3 of them did not fill out the VAS scales and I can’t find that mentioned in the paper anywhere.
Yea, obviously I noticed that too. But I don’t think it means something out of the ordinary was going on. In biomed it is the exception, rather than the rule, to explain the missing data.
Usually you detect it because they report something like “n = 10-13 mice per group.”
Well, there are probably good reasons to leave out those other mice, but those reasons are key info someone could use to replicate your study.
I agree. I submitted this to the blog here when it came out. We had a discussion about it. https://statmodeling.stat.columbia.edu/2019/10/28/what-happens-to-your-metabolism-when-you-eat-ultra-processed-foods/
I still think its the best diet study I’ve ever seen by a long shot. The real interesting results were that people appeared to eat an amount consistent with satiety after a certain quantity of protein, and that people ate higher quantities of ultra processed foods than raw foods.
This got me looking into the nutritional labeling we all see on the packaging (and also used for this and many other studies):
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/guidance-industry-guide-developing-and-using-data-bases-nutrition-labeling
So when you see 100 g of carbs on the label (or in the database) it means at least 80 g need to be present to be in compliance. But if you see 100g of fat that means no more than 120g can be there and remain compliant.
The main problem with the study is that the menus are not matched as well as it looks. IIRC the energy density of the processed food is like 2x that of the unprocessed food. They put like 3-5 glasses of crystal light on the table with the processed food to bring the overall energy density of the processed meal (incl. the calorie-free beverages) down to the level of the unprocessed menu. It’s kinda absurd when you look at the pictures. Other than that, I do like the design and there need to be more studies like it.
If that 0.3 is a standard error, the p-value cannot be 0.009:
observed_t_val<- 0.9/0.3 # 3
2*pt(3,df=19,lower.tail=FALSE) # 0.007361724
Probably these numbers are not accurately reported. How do we know that 0.3 is the SE?
Shravan:
Rounding error. “0.3” could be 0.32 or whatever.
RIght, I’d always check from 0.25 to 0.3499999999. It’s interesting that you call it rounding error. I’d only use that for when the author made a mistake in rounding, like truncation.
BTW, I know what you mean by rounding error Andrew, I just wouldn’t have thought to call it that in this kind of case. I suppose it still goes to me thinking of error as a human foible and words like noise about data.
I think sometimes that particular wording is one that can divide the uninitiated in stats speak when they read criticism. They’ll take any statement with “error” in it as an accusation.
Psyoskeptic. “Rounding error” is absolutely the standard terminology in numerical analysis to mean “the difference between the mathematically correct value accurate to infinite digits and the computers represented value”. It’s not a mistake by a human at all.
I agree with you though that a lot of people are thrown by the use of “error”. I’m so used to it that it doesn’t make any impression at all, others might be shocked to hear that there’s an **error** oh no how scandalous!