Plan 9 from PPNAS

[cat picture]

Asher Meir points to this breathless news article and sends me a message, subject line “Fruit juice leads to 0.003 unit (!) increase in BMI”:

“the study results showed that one daily 6- to 8-ounce serving increment of 100% fruit juice was associated with a small .003 unit increase in body mass index over one year in children of all ages.”

No confidence intervals but obviously this finding is very worrisome. Children shouldn’t be gaining weight.

Meir continues:

Of course it’s not a coincidence that it’s weird. I send you a very unrepresentative sample of the stuff I read. I mostly don’t send you ordinary schlock but rather things that are really weird – like a “0.003 unit increase in BMI” which is not only statistically insignificant but even if it was able to be substantiated would be of 0 health consequences.

I really enjoy seeing things like this, they are so ridiculous they are like those cult movies that are so bad they’re good.

P.S. Yeah, yeah, I know that this particular piece of junk science didn’t appear in PPNAS. But until PPNAS apologizes for wasting the world’s time with air rage, himmicanes, ages ending in 9, etc., I think we have the moral right to continue to use them as shorthand for this sort of thing.

20 thoughts on “Plan 9 from PPNAS

  1. Obviously, believing in or getting worked up about a BMI change of 0.003 is inane, and points to the usual problems of “binary” thinking (an effect exists or not) and not understanding noise. It’s great that you’re writing about this.

    However, your title and P.S. have the same flaws. Here, for kicks, are the titles of about 1/3 of the articles in this week’s PNAS. (I could copy and paste more…) What fraction do you think are “junk?” What’s your threshold for maligning the whole journal, or using it as a cute shorthand for crap? This isn’t an issue of social vs. natural sciences. (In fact, one of the worst articles I’ve ever read was a physics paper in PNAS, last year.) Nor is is that I particularly care about PNAS. It’s the same issue as raised above, and I think your excellent points in this and other posts aren’t well-served by the “PPNAS” schtick.

    Cu metal embedded in oxidized matrix catalyst to promote CO2 activation and CO dimerization for electrochemical reduction of CO2

    Enhanced interplanetary panspermia in the TRAPPIST-1 system

    Broken flow symmetry explains the dynamics of small particles in deterministic lateral displacement arrays

    Ubiquitin S65 phosphorylation engenders a pH-sensitive conformational switch

    Pharmacology of the Nav1.1 domain IV voltage sensor reveals coupling between inactivation gating processes

    Peripheral modifications of [Ψ[CH2NH]Tpg4]vancomycin with added synergistic mechanisms of action provide durable and potent antibiotics

    Intraoperative assessment of tumor margins during glioma resection by desorption electrospray ionization-mass spectrometry

    Subsurface oxide plays a critical role in CO2 activation by Cu(111) surfaces to form chemisorbed CO2, the first step in reduction of CO2

    Age of Jupiter inferred from the distinct genetics and formation times of meteorites

    Interface-induced multiferroism by design in complex oxide superlattices

    Evaluation of a proposal for reliable low-cost grid power with 100% wind, water, and solar

    Immunoengineering nerve repair

    Cas1 and the Csy complex are opposing regulators of Cas2/3 nuclease activity

    Selective oxidation of aliphatic C–H bonds in alkylphenols by a chemomimetic biocatalytic system

    Network dynamics of social influence in the wisdom of crowds

    Human genome-wide repair map of DNA damage caused by the cigarette smoke carcinogen benzo[a]pyrene

    Crystal structures and atomic model of NADPH oxidase

    Biomechanical coupling facilitates spinal neural tube closure in mouse embryos

    How temporal patterns in rainfall determine the geomorphology and carbon fluxes of tropical peatlands

    Maintenance of antiangiogenic and antitumor effects by orally active low-dose capecitabine for long-term cancer therapy

    Interplay between tolerance mechanisms to copper and acid stress in Escherichia coli

    Gel-forming mucins form distinct morphologic structures in airways

    • Raghu:

      My problem with PPNAS is not that it publishes crap. Every journal publishes crap; it’s unavoidable for a journal that publishes 3000 papers a year. My problem with PNAS is the hype. Part of this is the fault of journalists who think that anything associated with the National Academy of Sciences must be solid research. But part of this is the fault of PNAS itself, given that on its webpage it advertises: “PNAS publishes only the highest quality scientific research.” That’s what they claim, so I hold them to this high standard. If you think it’s unfair to “malign the whole journal,” just because they publish some really bad stuff, you might want to take it up with whoever is standing by the slogan, “PNAS publishes only the highest quality scientific research.”

      Or, to put it another way, my problem with journals is not that they publish bad papers; it’s that they’re part of a system that minimizes or even denies error, a system under which virtually no correction is enough for authors to accept that their conclusions might be mistaken. I pick on PPNAS because PPNAS contributes to and benefits from this system: they contribute via their “PNAS publishes only the highest quality scientific research” and their unwillingness to apologize for himmicanes, air rage, etc., and the benefit from all the typically uncritical media exposure given to their papers. The National Academy of Sciences can and should do better.

      • Or, to put it another way, my problem with this administration is not that they make mistakes; it’s that they’re part of a system that minimizes or even denies error, a system under which virtually no correction is enough for authors to accept that their conclusions might be mistaken. I pick on the White House because the White House contributes to and benefits from this system: they contribute via their “Make America Great Again” and their unwillingness to apologize for incorrect Tweets, false statements by press secretaries, etc., and the benefit from all the typically uncritical media exposure given to their events. The United States can and should do better.

        • Mark:

          i think it’s a good thing that we have media outlets holding the Republicans’ feet to the fire, and other media outlets doing the same for the Democrats. It’s not necessary for every journalist to offer both these perspectives; it’s just good for the criticism and responses to be based on facts and clear reasoning.

          PNAS is a little different because they’re not part of a two-party system. There’s no natural “opposition” to the National Academy of Sciences. Also, when criticizing PNAS, I’m not criticizing the National Academy of Sciences as an institution, I’m just saying they can do better by being more open to criticism.

          The political story is different, in that people are criticizing the Trump administration (and others were similarly criticizing the Obama administration, and before that the Bush administration, etc.) not merely for lack of openness to criticism but for particular bad or dangerous decisions. So I don’t think your analogy works so well.

        • I was just being flippant; as I was reading your (astute) point, I was thinking, “This sounds like someone is talking about Trump administration…” So I thought I’d make a (clever?) comment.

        • “i think it’s a good thing that we have media outlets holding the Republicans’ feet to the fire, and other media outlets doing the same for the Democrats.”

          GS: An alternative view is that the media fits nicely into the façade that “the government” is kept in check by either the media or the voters in some MEANINGFUL way – that is, in a fashion commensurate with problem. Put simply, but somewhat non-technically, all the self-congratulatory back-patting that goes on surrounding the supposed superiority of democracy and the role of a free-press etc., fits nicely into the overall aims of the propaganda arm of the controllers (the ultra-rich and the ultra-rich politicians that collude with them); *we* busy ourselves with the status quo while we slip inexorably toward totalitarianism. This is manifested in the Trump era; Trump’s antics are not really “the story” here. The real story is that he is President when he is so clumsy vis-à-vis the Grand Façade. Politicians are appalled that he is so blatant in his reverse-Robin Hood redistribution of the wealth. That was supposed to be discrete enough to be below “the threshold.” Government has but one, two-part function, 1.) keep the rich rich and 2.) keep the poor from killing the rich. As long as the correct balance is struck, the system can continue to slide toward totalitarianism without a hitch. Anyone that thinks that electing a mainstream republican or democrat significantly (in the real sense of “significantly”) alters the trajectory towards totalitarianism is, IMO, just as gullible as the Trumpies. Again, though, to get back to the point about the press in case it isn’t clear: The “free-press” is the real “opiate of the masses” because it legitimizes the non-Trump status quo which is, in fact, characterized by the inexorable flow towards totalitarianism just as much as Trump’s clumsy rampage.

      • That’s a fair point. It would be exhausting, though, to apply this standard consistently. I’d bet that Columbia makes laughable statements about how excellent it is, and that it benefits from a “system” of elite university status, but we don’t feel compelled to bring up its name whenever any university over-hypes itself.

        Even sticking to journals: the non-tabloid Physical Review Letters, for example, states that it “is the world’s premier physics letter journal. It publishes short, high quality reports of significant and notable results.” There’s certainly crap in PRL, and crap that gets media coverage. It would be perfectly fair to call them out on it, just as it’s fair, constructive, and admirable that you call out crap that’s in PNAS. However, I still maintain that it would be unfair to title a post about an awful physics paper published elsewhere “junk from PRL,” just because PRL benefits from “the system.” The same holds for a lot of journals.

        • Raghu:

          I think PPNAS should remove its statement, “PNAS publishes only the highest quality scientific research,” as it is evidently false.

          Your PRL statement doesn’t seem so bad: they say the publish “short, high quality reports.” That’s a lot better than saying “highest quality” and it’s a lot better than saying “only.” I don’t see any grounds for criticizing the PRL statement, and I similarly would be fine if PPNAS wrote that they “publish high quality scientific research” or even that they “aim to publish the highest quality” etc.

          If Columbia University says that it offers an excellent education, that would not bother me. If Columbia said that they admit only the highest quality students, or that they employ only the highest quality researchers, then, yes, that would bother me.

        • The impression I had even before I encountered Andrew’s PPNAS ranting was that PNAS has always (or at least for a long time) been in some ways like a vanity press — where, for example, a NAS member is almost always assured of getting his/her student’s work published, and often in a more timely manner than in other journals.

        • Martha:

          Could be. What’s weird is that PPNAS seems to have such high status among journalists. Go back a bunch of years ago, and I’d never heard of PNAS. I don’t really know what happened to transform PNAS from an obscure, archival journal of record to a high-profile, third only to Science and Nature in its ability to attract publicity.

  2. Sure, 0.003 per year may not seem like much, but when you’re looking in the mirror just a few short millennia from now, you’ll be singing a different tune.

  3. Why is it not considered a requirement that news articles written about published studies provide links to those studies? They link to everything else they can think of, no matter how tangentially related to the topic at hand. But linking to the paper that the article is about? Who wants that?

    I know I can google it, I just find this really annoying. Anyway, here it is:

    “RESULTS: Of the 4657 articles screened, 8 prospective cohort studies (n = 34 470 individual children) met the inclusion criteria. Controlling for total energy intake, 1 daily 6- to 8-oz serving increment of 100% fruit juice was associated with a 0.003 (95% CI: 0.001 to 0.004) unit increase in BMI z score over 1 year in children of all ages (0% increase in BMI percentile). In children ages 1 to 6 years, 1 serving increment was associated with a 0.087 (95% confidence interval: 0.008 to 0.167) unit increase in BMI z score (4% increase in BMI percentile). 100% fruit juice consumption was not associated with BMI z score increase in children ages 7 to 18 years.

    CONCLUSIONS: Consumption of 100% fruit juice is associated with a small amount of weight gain in children ages 1 to 6 years that is not clinically significant, and is not associated with weight gain in children ages 7 to 18 years. More studies are needed in children ages 1 to 6 years.”

    So the 0.003 is a z-score increase, and the CI doesn’t contain zero! But only if you exclude children over 6. Therefore future research should exclude children over 6. And this research needs to be performed, but by the way the effect is too small to be of interest.

  4. “Auerbach, lead author of the new study, said he is the parent of a 7-month-old boy who soon may be offered fruit juices in day care and at school.”

    Wow – when I was an elementary school student in the 80’s, all we were told to worry about being offered in school was crack.

  5. It’s even funnier if one considers the absurd unit of BMI, kg per meter squared. Maybe those children aren’t gaining area fast enough!

      • I’m pretty sure the cost-benefit units are the ones of interest – so we need to get to $ per kg-meter-second-millimeter of mercury to get the policy-relevant scaling.

        Of course, if they just did the right thing and switched to dimensionless analysis, we would get a nice and clear scientific answer like 4 or 17.

        • BMI * your shoe length squared * (volumetric unit price of fruit juice / density of water) /( monthly payment rate on medical school loans) / average shelf life of fruit juice

          I get on the order of 1e-4 so clearly this has no effect

  6. This is one of those times where since we know the nutritional content of fruit juice is crap, it reduces confidence in the study that the effect is so small. Of course, the normal practical limitations on researching whether a single food item makes a child gain weight more than they are already bound to gain makes it seem unlikely you’d ever get much more than noise unless there are unobserved variables that correlate positively with both weight gain and fruit juice intake (I suspect there aren’t).

Comments are closed.