Hey, Nature magazine! Reputation is a two-way street.

Kevin Gray pointed me to this everyday, run-of-the-mill, bread-and-butter bit of junk science, “Association between the screen time spent watching short videos at bedtime and essential hypertension in young and middle-aged people: a cross-sectional study.”

I heard once that doctors at many institutions are required to publish research papers as part of their professional advancement, and so it stands to reason that there will be millions of trash papers floating around. This one is bad, but not especially bad. If it were a student project, I’d just tell them to remove or qualify the causal language and to plot the raw data. As a contribution to the medical literature, I guess the paper has no value, but I also assume that contributing to the medical literature is not really the point, that it’s some sort of professional qualification exercise.

Anyway, it’s no big deal, just something that someone emailed to me, and I have no idea how he came across it. It’s not like anyone would ever hear about it, right?

Ummm, how exactly did my correspondent hear about this paper? Let me do a quick google news search:

Oh, shoot, it actually did get some press . . . that’s unfortunate.

It was also promoted by the publisher of the journal:

Here’s the journal’s webpage:

And this brings us to the subject of today’s post, which is not that silly article but rather its branding.

The journal is called BMC Public Health. BMC stands for BioMed Central, who describe themselves as “the pioneers of open access publishing.” They’re part of “Springer Nature.” I guess the way this works is that Springer paid for the Nature brand name, or something like that. Springer has the money, Nature has the reputation, so it’s a natural exchange.

As I’ve said before, reputation is a two-way street:

From one direction, Nature has that reputation, which rubs off on Springer Nature, which in turn lends credibility to journals such as BMC Public Health which use the Springer Nature branding, which in turn lends credibility to papers like “Association between the screen time spent watching short videos at bedtime and essential hypertension in young and middle-aged people: a cross-sectional study,” which in turn gets them into the press and into the hands of my correspondent.

From the other direction, we see a useless paper in a journal with the Nature brand, and this makes me respect things published in Nature a little bit less. Fair enough; Nature made money by selling their name, and nothing comes for free.

As with Matt Damon, Larry David, and LeBron James (all notorious for making money from endorsing a financial scam), so with Nature.

P.S. Dale in comments noticed that the paper includes this obviously false statement:

That’s just nuts! The whole paper is a data analysis.

Weighting of evidence and conflict of interest at the FDA and elsewhere

JJ Levine writes:

I wanted to share a thought about the expected changes at the FDA and related agencies in the new administration. I am wondering if many of the controversies in health are really a problem of data aggregation.

Here is an article from the Washington Monthly that talks about the new proposed head of the FDA and some of his controversial opinions.

Aside from one erroneous line (“In statistics, ‘significance’ refers to the likelihood that the results are correct, not that there is a major effect.”), the article seems to be mainly about how to put together evidence from various different studies.

What I am wondering is what can be done to foster decision making at the highest levels that is not based on pull quotes from selected studies but is based on the weighing of evidence from different sources? How would a Bayesian framework deal with these issues?

The linked article, by Merrill Goozner, quotes Marty Makary, the potential incoming leader of the FDA, as writing in his book “Blind Spots” that “Home deliveries triple the risk of infant mortality.”

Goozner follows up on this triple-the-risk statement:

That shocking claim . . . sent me rushing for the footnotes. There were none. So, I searched the medical literature and found a 2010 meta-analysis (an analysis using pooled data from a group of studies on the same subject) in the American Journal of Obstetrics and Gynecology that made that claim. It reviewed data from over one-half million planned hospital or planned home deliveries. However, only 1 of the 12 studies used for the meta-analysis was a randomized controlled trial. The rest were observational studies, some with some without matched cohorts. The low quality of the evidence was blasted by letter writers to the Journal.

I [Goozner also found a subsequent review of that meta-analysis that warned, “the authors’ conclusions should be treated with some caution as they did not reflect all the evidence presented in the review.” . . . the meta-analysis showed no statistical difference in the two groups in perinatal infant mortality (deaths up to 7 days after childbirth).” The alleged “tripling” was only in neonatal infant mortality (deaths up to 28 days after childbirth), which logic suggests would include more deaths not associated with childbirth itself. The critique also pointed out that the data on neonatal infant deaths came from trials that included fewer than 50,000 births — less than a tenth of the meta-analysis’ total review population.

In a long book, anyone can make mistakes, and Goozner writes that many of Makary’s points are very reasonable. Goozner is concerned with Makary’s blind spots:

While Makary digs deep in the medical literature to find studies that justify his opinions, he avoids mention of the hundreds of medical journal studies published throughout his quarter-century career that document the pernicious effects of industry’s role in financing medical research, clinical practice guidelines, patient advocacy groups, continuing medical education, and physician marketing. He ignores how industry-funded research supported the extensive use of drugs and devices beyond the indications included on the FDA-approved label, often causing great harm in the process.

At this point, I should note my own conflicts of interest, that I’ve collaborated with colleagues in the pharmaceutical industry and I’ve received research support in those collaborations.

I agree with Levine that combining information is key here. I don’t see any easy answers. The conflicts of interest in health care are huge. That said, we don’t just have to talk about health care here. The conflicts of interest in military procurement, funding of police departments and prisons, etc., are even bigger, right? And, as Goozner notes, political pressures and conflicts of interest regarding the FDA are not new.

For which cases does AI help with classification (medical diagnosis example)? “The narrow beach between the continent of clear effects and the sea of confusion”

Paul Alper points to this news article entitled “ChatGPT Defeated Doctors at Diagnosing Illness,” which reports:

A small study found ChatGPT outdid human physicians when assessing medical case histories, even when those doctors were using a chatbot. . . . doctors who were given ChatGPT-4 along with conventional resources did only slightly better than doctors who did not have access to the bot. And, to the researchers’ surprise, ChatGPT alone outperformed the doctors.

Alper writes:

In contrast, without any AI, over the summer I was (instantly) diagnosed as having a hernia, an exceedingly common condition, and conventionally operated on, unaided by AI. My path of recovery was exactly as Wikipedia (boringly?) predicted and I am now as physically healthy as I am, because of the election results, mentally a basket case. So, the issue might simply be that AI is of little diagnostic value when things are commonplace, i.e., straightforward, far from “exotic.” AI becomes valuable only when there are competing treatments and uncertainty.

Interesting point, and I guess this is a general issue with such classification problems, that most of the cases are easy, and the improvement is coming on the frontier: those cases where the classification is hard enough that the standard approach might get it wrong, but there’s enough potential information that a new approach can do better.

As we’ve said before, statistics lives on a phase transition, “the narrow beach between the continent of clear effects and the sea of confusion”: that sweet spot between signals that are so clear you don’t need statistics, and signals that are so weak that statistical analysis can’t help.

This reminds me of a similar thing we’ve discussed regarding models for forecasting elections and sporting events: if you can predict a continuous outcome, that’s great, but if you can only predict the winner, things are more tricky. Your best data come not from the blowouts (just about any model can predict these) or the nailbiters (the prediction here is basically random) but rather from the games or races with moderate margins of victory.

That last point relies not just on the properties of the estimation or forecasting method but also on the data-generating process. The relevant assumption is that the score differential (or vote differential) is generated by a continuous process, which then determines who wins. If there’s a way for a candidate or team to get a win in some direct way, without just improving their score or vote differential, then that’s another story. If there is no such way to get the win by bypassing the continuous process, then forecasts for very close outcomes will necessarily be essentially coin flips.

This is an interesting point to me, because the statistical properties of an estimator are usually taken to just depend on the process of estimation, but here they depend on the data-generating process as well.

To get back to the problem of evaluating diagnoses: it would make sense to focus on these not-too-easy, not-too-hard intermediate cases. Also, though, I guess that some doctors could miss the obvious ones too sometimes—everybody can have a bad day, right? So it could be that the greatest value in AI is in catching the super-obvious mistakes.

I’m not sure. Maybe there’s some literature on this?

Autism junk science: The only part of this story that surprises me is that the outside critic “found it hard to believe just how flawed it turned out to be”

Dorothy Bishop followed up on a story reported in the (London) Daily Telegraph as “Severe autism can be reversed, groundbreaking study suggests”—the headline was later changed to “Autism can be reversed, scientists discover” and then to “‘Autism can be reversed’, scientists claim.”

Bishop supplies some background:

The article, by D’Adamo et al., was published in a special issue of an MDPI journal, the Journal of Personalized Medicine. The special issue “A Personalized Medicine Approach to the Diagnosis and Management of Autism Spectrum Disorder: Beyond Genetic Syndromes” appears largely to be a vehicle for papers by the guest editor Richard E. Frye, who co-authored 3/4 editorials, 3/9 articles and 1/1 review in this collection. He was editor but not an author on the paper by D’Adamo et al, which is categorised as a “case report”.

Essentially, this is a description of development of a pair of non-identical twin girls who were diagnosed with autism at 20 months of age, along with various gastric and motor conditions, and were subsequently subjected to a remarkable list of interventions, viz:
· Parents worked with an autism parent coach who informed them about the link between “total allostatic load” and developmental disorders, a concept promoted by the Documenting Hope Project that one of the authors is affiliated with
· Parents accessed resources, including free webinars, through Epidemic Answers
· Parents took part in a parent forum called “Healing Together”
· Applied Behavior Analysis (ABA) from 22 mo to 33 mo
· Speech Therapy starting at 24 mo
· Rigorous diet and nutrition intervention eliminating sources of glutamate, following Reduced Excitatory Inflammatory Diet
· A strict gluten-free, casein-free diet that was low in sugar and additives
· Dietary supplements, including omega-3 fatty acids, multivitamins, vitamin D, carnitine, 5-methyltetrahydrofolate and “bio-individualized homeopathic remedies”
· Family consulted a naturopathic doctor who used IntellxxDNA genomics tool to recommend diet rich in tryptophan, vitamins B12, B6 and folate, betaine and choline, lion’s mane mushroom and resolvins, as well as some dietary variants specific to each twin.
· Neuro-sensory motor reflex integration given by an occupational therapist
· Environmental evaluation of the home for air quality, mould and moisture, culminating in a visit by a Building Biology Environmental Consultant who identified possible water damage.
· Cranial osteopathy, as recommended by a developmental optometrist.

When first diagnosed, the girls were given an assessment called the Autism Treatment Evaluation Checklist, which was repeated after 18 months. Scores declined (i.e. improved) for both girls, with one twin improving substantially, and the other less so.

I haven’t been able to find any norms for the ATEC, but there’s a bit of data on trajectories in a study by Mahapatra et al (2018) and this shows that scores normally improve between the age of 2 and 3 years. We may also note that the ATEC is completed by a parent, and so may be influenced by parental expectations of improvement.

On this blog we sometimes talk about “bad science”—this would be work that generally follows the methods of science but has some serious flaws in design, measurement, or analysis, with the most common problem being measurements that are too noisy to allow any realistic effects to be discerned from the data—and we talk about “junk science,” which could be defined as bad science as a general practice. That is, we say that researchers are doing junk science when they set up a research program that allows them to produce a stream of bad science.

For example, the notorious beauty-and-sex-ratio paper is bad science—it’s a study that had no chance of revealing anything useful, it was just an exercise in noise-mining that then was published and hyped—, and I’d describe the series of papers on the general topic published by that researcher as being junk science, in that they were a systematic production of bad science studies. Similarly, the paper on governors’ life spans was bad science; the general approach of drawing big claims from statistical significance in regression discontinuity analysis is a particular form of junk science.

Beyond bad science and junk science is what we might call “fake science”: reports that have the superficial form of science but do not involve any aspects of scientific inquiry. This could include fake data, plagiarism, or completely empty word salad, or various versions of data misrepresentation, politically motivated hackwork, or extreme p-hacking.

Based on Bishop’s post and a look at the paper itself, I’d characterize this autism paper as fake science: It’s written up like a scientific paper and it’s published in a journal that at first glance looks like a scientific paper, it uses scientific jargon (“A parent-driven, multidisciplinary, therapeutic intervention involving a variety of licensed clinicians focusing primarily on addressing environmental and modifiable lifestyle factors”), but ultimately it does not seem to be a serious scientific investigation. But if you want to be more charitable you could just call it bad science, for the usual reason that it makes big claims that are not supported by the data at hand, and you could call it junk science, in that it’s part of this iffy journal issue that Bishop discusses.

As Bishop writes:

The proliferation of anecdotal reports in the literature gives ammunition to those who wish to promote all kinds of unevidenced treatments: they can point to these “peer reviewed” papers as evidence of scientific respectability. . . . every practitioner with a dubious product to sell will see this journal as a perfect outlet for their promotional material.

The biggest mystery to me is how this ended up in the Daily Telegraph. How did their reporter hear about this little paper published in an obscure journal? I guess that someone involved with the paper or the journal has an excellent publicity department.

Finally, I’m surprised that Dorothy Bishop, after reading the paper, “found it hard to believe just how flawed it turned out to be.” Bishop is a longstanding observer and critic of junk science. She’s seen so much bad stuff, that I’m surprised that she’s so surprised!

I agree with her that this sort of thing is horrible, both in itself and in how it degrades the general world of science and science communication. If people want to come up with theories, tell stories, and share anecdotes, they should do so without cloaking it in the language of science.

Hey! Here’s what to do when you have two or more surveys on the same population! (Combining survey data obtained using different modes of sampling)

This has come up before (in 2011 and in 2018) but someone just asked me again today so I thought I’d give this advice again, since I haven’t seen it written anywhere.

Here was the question:

I’m reaching out to seek your advice on how to integrate two probability samples for the new Poverty Tracker cohort.

In the Poverty Tracker 2024 cohort, we finally got the chance to include an Address Based Sampling (ABS) sample. Our sample design includes half of the sample coming from the Random Digit Dial frame and the other half recruited through the Address-Based Sampling. I’m trying to map out how we’d be able to integrate those samples from two frames but having a hard time to find resources.

And here’s my reply:

The right thing to do is to simply pool the data together from both samples into a single dataset. Also in the dataset include an indicator that says, for each data point, which sample it came from. Then we do all analysis (including construction of weights, if we want to do that) using the combined dataset. When fitting models, running regressions, we include the indicator for which sample, just in case it is predictive of anything.

You can do something similar if you want to combine more than two samples; just include an indicator for each sample. And the same idea applies when combining raw data from multiple surveys (although then you might need to do some work to line up relevant poststratification variables, for example if the two surveys use different categories or different question wordings when asking about education or ethnicity or party identification or whatever).

The oldest famous person

Reading this obituary of the oldest major league baseball player (a former Yankee and Oriole, he died at the age of 100) made me think again about the question of who was the oldest famous person.

When this came up last year, I gave the nod to George Abbott, who directed On the Town, wrote and directed Damn Yankees, and was involved in a bunch of other Broadway classics–and lived to 107. There were a few famous people who lived to 104, including Rose Kennedy, Olivia de Havilland, Beverly Cleary, and Oscar Niemeyer.

I was curious who are the oldest people currently alive. There’s a wikipedia page on it. The oldest name on this list who has any fame at all (other than being famous just for living so long) is Marita Camacho Quirós, former First Lady of Costa Rica. She’s 114. There are 50 people on that list and she’s the only famous one.

I still want to make a graph mapping out the efficient frontier of age and fame.

P.S. Given the approximately six-month delay on this blog, there’s a pretty good chance that Marita Camacho Quirós will no longer be around when this post finally appears. I’m rooting for you to keep going, Marita!

P.P.S. Also relevant to the discussion is this article by Michael Kinsley from 2008.

“You can cite peer-reviewed research in support of almost any claim, no matter how absurd.”

Adam Marcus and Ivan Oransky write:

Twice during his Senate confirmation hearings at the end of last month, Robert F. Kennedy Jr., America’s new health secretary, brought up a peer-reviewed study by . . . Anthony Mawson, an epidemiologist and a former academic who has published several papers alleging a connection between childhood vaccines and autism. (Any such connection has been thoroughly debunked.)

But wait! you might ask. If this has been pre-debunked, how did it appear in a peer-reviewed journal? Marcus and Oransky explain:

[The paper] to which Kennedy was referring, appeared in a journal that is not indexed by the National Library of Medicine or by any other organization that might provide it with some scientific credibility. One leading member of the journal’s editorial board, a stubborn advocate for using hydroxychloroquine and ivermectin to treat COVID-19, has lost five papers to retraction. Another member is Didier Raoult (whose name the journal has misspelled), a presence on the Retraction Watch leaderboard, which is derived from the work of a nonprofit we cofounded, with 31 retractions. A third, and the journal’s editor in chief, is James Lyons-Weiler, who has one retraction of his own . . .

Perhaps a scientist or politician—and certainly a citizen-activist who hopes to be the nation’s leading health-policy official—should be wary of citing anything from this researcher or this journal to support a claim. The fact that one can do so anyway in a setting of the highest stakes, while stating truthfully that the work originated in a peer-reviewed, academic publication, reveals an awkward fact: The scientific literature is an essential ocean of knowledge, in which floats an alarming amount of junk.

Yeah, someone had actually pointed me to that paper a couple weeks ago and I responded that it’s not worth looking at something published on a propaganda website. With enough effort it should be possible to find the flaws in the paper, but I don’t want to be drawn into a game of whack-a-mole, as unscrupulous and clueless people (it can be hard to distinguish between the two; recall Clarke’s law) can produce b.s. faster than I can clean it up.

At this point, you might be tempted to say, “Hey, stop gatekeeping, man! Just cos something is published in Pravda, that doesn’t make it false. Evaluate the claim on its own merits.” To which I’d reply: sure, but in this case the box is wide open. I take a paper published in a propaganda website no more seriously than an unpublished paper on Arxiv, or Vixra, or some student’s term paper, or this stuff.

The point is, if you want to be open-minded, there’s no good reason to start with work promoting a politically loaded, much-debunked idea and published on a propaganda site.

Unfortunately, the legitimate peer-review system also produces its share of duds, including papers by decorated academic researchers publishing in top journals–that last guy is a Professor of Ethics, if you can believe it! The situation is grim.

Controversy over different estimates of covid origins probabilities

John Williams writes:

Here’s a new zombie idea.

Claims that the Wuhan lab is the likely source of SARS CoV2, and that Anthony Fauci squelched inquiries into this to please the Chinese, have reared up lately in places not normally given to right wing hysteria, for example a recent New Yorker article by Daniel Immerwahr (Doctor’s Orders). The clear consensus of qualified scientists, expressed most recently in a new World Health Organization report, is that the available evidence points to a zoonotic spillover at the Wuhan wet market as the most likely source, although the lab leak hypothesis can’t be ruled out. I don’t expect this report to change any minds, however.

This is a great topic to bring up on the blog, given that a few months ago we discussed an two articles, one by physicist Michael Weissman and one by economist Andrew Levin, both arguing that there are strong odds in favor of the lab leak hypothesis.

In contrast, the World Health Organization report concludes:

The work to understand the origins of SARS-CoV-2 remains unfinished. . . . while a zoonotic origin with spillover from animals to humans is currently considered the best supported hypothesis by the available scientific data, until requests for further information are met or more scientific data becomes available, the origins of SARS-CoV-2 and how it entered the human population will remain inconclusive.

So there’s real scientific disagreement here! Two groups studying the same data, one group saying the evidence isn’t clear (the best they can say is that one hypothesis is “currently considered the best supported by the available data”) and the other saying the evidence in favor of the other hypothesis is overwhelming.

What’s going on?

When this came up in February, I wrote that I didn’t know what to think:

It’s hard to compute Bayesian probabilities for this problem, not so much because of the priors but because of the likelihood, which is the probability of the data given the model. One problem is the selection of what is considered to be data, the other problem is that the model (“lab leak” or “wet-market leak” or whatever) is not clearly specified–in statistics jargon, these are “composite hypotheses.” This is not a criticism of this particular papers per se; it’s just a general difficulty with this sort of analysis. It’s not clear to me that Bayesian inference is the right way to attack this sort of problem. But I’ve been intimidated by the technical biological details in all these analyses so I haven’t looked at them personally.

For example, there’s this from Weissman’s article:

A natural outbreak could come from any of a diverse collection of pathogens, but this outbreak matched the specific subcategory of virus being studied in the Wuhan labs. Another update will come from a special genetic sequence that codes for the furin cleavage site (FCS) where the UNC-WIV-EHA DEFUSE proposal suggested adding a tiny piece of protein sequence to a natural coronavirus sequence. The tiny extra part of SC2’s spike protein, the FCS that is absent in its wild relatives, has nucleotide coding that is rare for related natural viruses but seems less peculiar for the most relevant known designed sequences, the mRNA vaccines.

I don’t know viruses, and this sort of thing is just impossible for me to judge.

Similarly, there’s this from the WHO report:

One of the viruses currently known to be closest to SARS-CoV-2 (RaTG13) was described along with an original characterization of SARS-CoV-2 by Zhou et al. (Zhou et al., 2020b) An addendum to this paper mentions that the virus material had been available since 2012/2013 at the Wuhan Institute of Virology in the form of an uncultured original animal stored sample (ID4991) with published partial sequence characterization in 2016, amended to near full sequence characterization in 2018, renamed RaTG13 . . . After the first identification and sequencing of SARS-CoV-2, comparison of the partial sequence led researchers to complete sequencing of the full genome of RaTG13. However, the RaTG13 genome has a low similarity with SARS-CoV-2 in the critical RBD of the spike protein (only 11 of 17 receptor contact residues are conserved).

The similarities and differences between these viruses must be very relevant to the probability of a lab virus becoming COVID, but what do I know about these viruses? Absolutely nothing! So right there I run into a wall. Which is kinda scary because I think I’m pretty good at assessing evidence, and if I’m so stuck here, what would that say about the ability of other non-experts to figure this out?

Usually the way to go is to rely on the experts, which in this case is the WHO report, so I guess my inclination is to go with that for now. But then I have to be clear that this is why I’m relying on it, because I find it difficult to otherwise judge. And, indeed, the WHO report itself cites other authorities who are equivocal:

On 2 April 2025, the French National Academy of Medicine published its report entitled ‘From the Origin of SARS-CoV-2 to the Risks of Zoonoses and Dangerous Virus Handling’, which analysed two hypotheses: 1) SARS-CoV-2 is a zoonosis and 2) the origin of SARS-CoV-2 is linked to a laboratory accident. The Academy did not reach any firm conclusions on the subject, which it has taken as a starting point for reflection on possible recommendations and proposals for action to better anticipate and react to emerging diseases.

So, uncertainty!

In theory I guess that some qualified epidemiologist could reconcile Weissman’s report and the WHO report. Such reconciliations are possible, but it can take some work and it might not attract much interest; for example, our paper, Reconciling evaluations of the millennium villages project, has been cited only 2 times since its publication in 2022. Fair enough–there’s not so much interest any more in that sort of international development project–; still, a bit frustrating. Given that Weissman’s paper itself remains unpublished, it’s not clear that someone with the qualifications to evaluate all this evidence would want to put in the time and effort to perform the reconciliation.

The job could be done, I’m sure. Nick Brown and I did something of the sort in our recently published assessment of certain published claims of mind-body healing, and that indeed took a lot of work on our part. You’d just need someone with a similar temperament and reputation to Nick and me, along with the necessary biology expertise.

P.S. I looked up the other John Williams, and . . . he composed lots of soundtracks before he did Jaws. Lots and lots! His first soundtrack was in 1954, he did a couple of the Gidget movies, a bunch of TV movies, the score for Fiddler on the Roof, the Poseidon Adventure, all sorts of things. I had no idea. Quite the career.

“Flipping the Narrative in ‘Slouching Towards Utopia'”: Counter-narratives going beyond the default economics model of exponential growth

The above graph is an amusing example of futurologists falling in love with the idea of exponential growth. It was reprinted by historian Sean Manning in a long essay in the form of a blog post, on the difference between how historians and economists think about economic history.

You’ll have to read the whole thing–also it’s written as a review of a book, “Slouching Toward Utopia,” by economist Brad DeLong, that I haven’t read–but here’s Manning:

Slouching Towards Utopia is . . . a big ideas book, or as DeLong prefers to call it a grand narrative. . . . Almost any paragraph of this book poses two or three fun subjects for debate. But this week, I [Manning] will put most of them aside and show how this book tells one story while the author wants to tell a different one.

A Grand Narrative

Slouching speaks of things which were, and things which are, and things which will be. . . . DeLong is an economist, and economists love to model things as exponential growth. So DeLong makes up numbers for world population and GDP in 6000 BCE, 1000 BCE, 1 BCE, and 1500 CE . . .

Slouching ends in the recent past, but it is optimistic about the future. Incomes are no longer increasing as fast in the rich countries, but much of the world is still poor, and prosperity in the rich countries which grows at 0.5% a year instead of 2% a year is still growth. “There is every reason to believe prosperity will continue to grow at an exponential rate in the centuries to come” (p. 11)

This is a grand narrative about incomes and population and exponential growth, three of economists’ favourite things. It is a narrative where before 1870 population grows slowly and prosperity barely grows at all, then from 1870 to 2010 both grow rapidly, and after 2010 prosperity is likely to continue to grow while populations stabilize or shrink. Economists love to tell grand narratives about trends, even though they know why historians have trouble believing in them. I have not written a book with a grand narrative. But I can summon the old magic of Max Weber and use the facts in this book to tell a different story.

A Counter-Narrative of What Was

I [Manning] don’t see much value in estimating the population of the world in 6000 BCE when we can’t agree on the population of the Americas in 1492 within a factor of 5 [Manning’s original post said “a factor of 20” here, but after looking back at the literature he changed this to a factor of 5. — AG], and took decades to agree on the population of the Roman empire under Augustus within a factor of two. But we do have data on population and prices in specific places such as England since Domesday Book or Achaemenid Babylonia. These show cycles of rising and falling, with rising population tending to correspond to falling wages and vice versa. In the best times, an ordinary worker or peasant might consume twice as much as the worst times.

I think most people could agree that populations tended to slowly increase, particularly when new crops or methods of farming were introduced. . . . But its not clear that standards of living trended upwards. . . . data before the 19th century shows standards of living rising and falling in cycles, not stable with a gradual upwards trend. . . .

In past societies with great material wealth, such as the Roman empire, its not clear to me that the health benefits from tiled roofs or plastered aqueducts outpaced the health dangers of crowding people into cities and moving them back and forth. Height is a good proxy for health and nutrition, and height in Europe also goes up and down before the 20th century, rather than having a clear trend. Notably, it is higher in the material-poor early middle ages than the material-rich Roman empire or nineteenth century.

A Counter-Narrative of What is and What Will Be

Beginning in the 19th century, population and welfare begin to rise at the same time across much of the world. But countries which tried to imitate the richest countries often failed, and growth in any one country often stalled. . . . Economists can calculate, but physicists can too. Colleagues like University of California physics professor Tom Murphy have done the math . . . Economic growth that is decoupled from energy consumption won’t look much like GDP, and there are only so many hours in the day to enjoy our futuristic entertainment centers or teleport to our beach huts in Maui and stroll on the beach. Even if we avoid ecological catastrophe, a devastating plague, exhaustion of resources, and the replacement of humans-as-we-know them with something else, growth in prosperity is likely to end within the next century or two because of physical limits.

Manning concludes:

As a historian, I don’t think that exponential growth is the natural state of things. I think in terms of cycles and tradeoffs and phase changes. The simultaneous rise in health, wealth, and population in the long 20th century was a new thing in world history, and much of the world is transitioning into a new state. I think that looking at the past with a mind trained on the long 20th century is a mistake, and so is imagining the future that way. . . .

I think that [De Long’s] book would have been better if it presented exponential growth as typical of the long 20th century, not as the way economies usually behave.

Interesting. Slouching Toward Utopia also has a blurb from Paul Krugman, so maybe he and De Long can respond in comments.

The issue of exponential growth in the economy came up last year in our post, How to think about the claim by Justin Wolfers that “the income of the average American will double approximately every 39 years”?, where economist Wolfers shared this graph:

And here are some earlier posts regarding historians’ efforts to acknowledge historical uncertainty in social science and journalism:

History, historians, and causality

Uncertainties regarding historical facts

Decorative statistics and historical records

Uncertainties regarding historical facts

P.S. Manning’s counter-narratives can be given a “left” or “right” political spin. From the left, Manning’s arguments push against the right-leaning view that, whatever happens, the existing system of capitalism will bail us out: maybe a steady state of economic expansion will not be expected even in the absence of government interference. From the right, Manning’s arguments push against the left-leaning view in which gradually increasing prosperity is a given, and so our society can afford economic redistribution.

To put it another way, both the left and the right have contradictions in their views of the economy. Economic leftists oscillate between saying that the capitalist system is failing and needs to be replaced before we all choke on our own filth, and saying that, as the richest society in the world, we should be able to afford to massively reduce poverty and inequality. On the other side, economic rightists are torn between viewing capitalism as an unstoppable force (the resonant phrase is “there is no alternative”) and the idea that it is a fragile golden goose whose supply of eggs is endangered by any reforms.

In saying this, I’m not trying to “both-sides” the debate or imply that leftist and rightist views on economics are equally correct–it would be more accurate to say that I have various policy preferences on specific economic issues but I don’t have any coherent view of macroeconomics myself. As a political scientist, I’m interested in the arguments used by different sides in these debates, and I’m interested in how various ideas and talking points can implictly depend on the understanding of historical claims that themselves are in legitimate dispute.

P.P.S. More from Manning here, where he lists some factual errors or dubious claims in Slouching Towards Utopia, a book which Manning seems to like overall.

A bright shiny object: Misapplication of Benford’s law published in health economics journal

Benford’s Law is fun; it also is the source of endless confusion. Math is hard!

This came up recently when Kevin Lewis pointed to this paper, which states:

We use Benford’s law to examine the non-random elements of health care costs. We find that as health care expenditures increase, the conformity to the expected distribution of naturally occurring numbers worsens, indicating a tendency towards inefficient treatment. Government insurers follow Benford’s law better than private insurers indicating more efficient treatment. . . .

This sounded interesting, but I’m pretty sure they’re doing it wrong, because they try to evaluate the fit to Benford’s law within each “price bucket” ($100-999, $1000-9999, $10,000-99,999, and $100,000-999,999). Based on my understanding of the processes underlying Benford-like behavior, you wouldn’t necessarily expect the pattern to occur within each bin in that way.

Here’s an example of how things go wrong. The authors write:

We also follow Drake and Nigrini (2000) by calculating the mean of absolute deviations (MAD) to use as a way to assess conformity to the expected distribution. . . . We find that at the MAD for the first bucket of charges (0.010) shows a marginally acceptable conformity to Benford’s law. However, for the second (0.023), third (0.049), and fourth (0.092) buckets the MAD is greater than 0.012 indicating nonconformity. As expected, the MAD increases with the level of total-charges. . . . An additional possible explanation for this finding is hospital pricing strategies . . .

OK, here’s the problem. Here are the data:

This should really be a graph, but let’s not worry about that right here.

Let’s focus on the fourth bucket, because that’s where the discrepancy is highest. You see what’s happening, right? In that fourth bucket, we’re up there in the tail of the distribution, the tail is dropping fast, so, yeah, there are very charges over $200,000. That doesn’t mean anyone’s cheating in their billing; it’s just what you’d expect to see in the tail of the distribution. Benford’s law applies when the underlying numbers come from a distribution with have a wide dynamic range, and by binning in this way you’re destroying that.

I’d say I’m surprised this got published in a legitimate journal, but, you know, the problem with peer review is the peers. Everyone’s doing the best they can, Benford’s law is a bright shiny object, it gets misused just like linear regression gets misused, just like logistic regression gets used, just like hypothesis testing gets misused and misused and misused. The Benford example is just a little bit more interesting because the math confusion is something a bit less familiar than the usual statistical mistakes we see every day. Hence why I bothered with this post.

NIH plan to remove ideological influence from science. How does this fit in the junk science being promoted by the U.S. Dept of Health and Human Services?

A couple weeks ago we discussed the ridiculousness of the government mandating so-called gold standard science while at the same time promoting fraudulent, debunked, and flat-out fake research. We followed up the next day with another example.

A new plan from NIH

Since then I came across an interesting new development of a more positive kind. The director of the National Institutes of Health writes:

We’re working to remove ideological influence from science. NIH funding must be based on provable, testable hypotheses, not ideological narratives. Projects that don’t meet that bar are discontinued so we can focus on rigorous, impactful research.

I don’t think we could or should remove all ideological influence from science. The people who care about a topic enough to research it will often have ideological motivations. The ideological influence is there. That doesn’t mean it’s bad science.

Consider the field of criminal justice. I once talked with someone who was doing research on criminal justice who was, for ideological reasons, in favor of privatized prisons. Another time I worked with a colleague on a project involving the death penalty, which, for ideological reasons, he opposed. (I supported the death penalty at the time, but I have to admit that after completing that research project, my views changed: I continue to support the death penalty in theory but I don’t support it in practice, at least not how it’s done in the United States.) Anyway, my point here is that . . . this is fine. In some fields, ideological motivations and influence are unavoidable; the topic is still worth researching, and we just need to evaluate it carefully. Here’s another example, this time on the politically-charged topic of international economic development aid.

This does not mean that I think that we should let ideologically-motivated bad research off the hook. For example, the study discussed here seems useless to me, and I’m guessing that it was political ideology that got it published in Lancet. And don’t get me started on this story.

That said, I’m generally supportive of the NIH policy to reduce, if not to work to fully remove, ideological influence from science. Oftentimes it seems that ideological narratives are positively encouraged, as in the series of papers published in the Journal of Theoretical Biology that I criticized a few years ago: these were absolute junk and seem to have been constructed and promoted as part of an ideological project. Or that whole Implicit Association Test debacle. I’m not proposing that NSF necessarily give zero funds to such research–ideas can sometimes be revived–but, yeah, if they do fund it, it should be because it ultimately can improve health, not just because it fits an ideological narrative.

I’ll just add one more technical comment, which is the “provable, testable hypotheses” thing. Lots of hypotheses can’t be proved but there is still value in partial knowledge. Indeed, I have a problem with the whole attitude of expecting certainty from science: this leads to a roller-coaster of hype and disappointment. So I’d recommend removing the “provable” bit from that phrase. For most of science, “proof” is a snare and a delusion.

How does this fit in with the junk science being promoted elsewhere by the government?

The National Institutes of Health are part of the U.S. Department of Health and Human Services, and the head of that department has been promoting junk biological science such as, most notoriously, a discredited paper on vaccines and autism. They also released a report on children’s health that was full of fake citations. Others in the government have been promoting unsupported claims of widespread election fraud. All of this is ideological narrative, the opposite of “testable hypotheses” or “rigorous research.” Unfortunately, much of this junk science seems to have been “impactful” on policy, but impact is a bad thing if it’s unsupported and debunked claims that are making the impact. Indeed, this is a well known problem with bad science, that it can remain impactful long after it’s been discredited.

I can’t blame the director of NIH for the behavior of his boss at HHS. I do think there’s an awkward tension between the NIH attempting to remove ideological influence from science at the same time that its parent organization is putting ideological influence into public health decision making.

I don’t know how this gets resolved: maybe the executives at HHS are too busy with politics to worry about what’s happening inside the NIH? For political reasons, I’m guessing that the NIH director won’t call out the junk science being promoted by his boss. I do that sort of thing at Columbia, as here and here, but I’m just a statistics professor, not a political appointee. It’s gotta be a tough position to be running NIH, trying to remove ideological influence from science, while dealing with a boss who’s chest deep in the stuff. A difficult balancing act for sure.

P.S. Also I’d appreciate it if the NIH director corrects his misrepresentation of what I wrote. I mean, sure, I know he’s too busy to do this, indeed I would not want him to be spending his scarce time on something so minor. But the NIH employee who posts to that twitter account could issue a post with a quick correction and apology? It would only take a couple minutes, right?

Survey Statistics: 3 flavors of survey weights

Thomas Lumley (in his blog Biased & Inefficient) and Andrew, Jennifer, and Aki (in their book Regression and Other Stories) write about 3 flavors of weights in statistics:

1. survey weights describe how the survey sample can be scaled up to the population. One way to think about them is how many people in the population each person in the sample represents. These may be more complicated than 1/probability-of-being-sampled, as we will see below.

2. frequency weights describe how many actual observations have a certain pattern of variables (e.g. a weight of 12 on the women-age-37-from-New-York row says that there are 12 such women in the data).

3. precision weights describe the precision (1/variance) of the observations, making them quite distinct from the first 2 flavors.

For most point estimates, all 3 are treated the same. What’s different is variance estimation. Go read Thomas and Andrew et al. for lots more. We here will focus on the first flavor and break it down into 3 subflavors (my spell check says this is not a word) of survey weights:

A) inverse-response-probability weights are based on a model for response R (e.g. Logistic regression). The weight is then W = 1/Ehat[R | X]. See Lumley 2010 Section 9.2.2:

B) equivalent weights are based on a model for outcome of interest Y. We calibrate to known population data about X: E[Ehat[Y | X, sample]]. Then the equivalent weights are the expressions in Gelman 2007, at least for linear models of Y. Simplifying to classical linear models, this is the Generalized REGression (GREG) estimator, and is the same as our 3rd subflavor:

C) calibrated weights are based on finding W such that the weighted estimates of X match their known totals. There is no explicit model here.

Särndal 2007 contrasts B and C:

Statisticians who work in the area are of two types: Those dedicated to “GREG thinking” and those dedicated to “calibration thinking”….I am not venturing to say that the latter thinking is more prevalent in national statistical agencies and the former more prevalent in the academic circles, but perhaps there is such a tendency….The linear GREG estimator implies weights that happen to be calibrated…

Särndal got to be both types of statistician, as a researcher at Sweden’s national statistical agencies and as a professor.

Which type of statistician are you ? What’s your favorite flavor of weights ?

Here’s what’s happening with time-shifting of births.

A couple weeks ago we discussed the question, “What day of the year will have the fewest noninduced births?”

It was an interesting difference between the reasoning of a mathematician (who focused on one factor–the loss of an hour in the day when Daylight Saving Time begins–while ignoring lots of other factors) and statistical reasoning (where we start with the data and see what we can find). As some commenters noted, ultimately we want to use both forms of reasoning, with statistical analysis backed up by mathematical modeling. From a “sociological” point of view, I found it interesting that the mathematician focused on such a minor aspect of the problem, it just happened to be the aspect that was most suitable to direct mathematical analysis.

Here’s the birthday analysis that Aki and I did:

which is fine, but it’s all births, not just noninduced births. In my post, I pointed to a paper that separated the data by natural, C-section, and induced births but only had showed data for thirty days of the year:

The paper said the numbers came from the National Center for Health Statistics, but I wasn’t able to find data on births by date on their website, so I sent an email to the authors to see if they the counts for all three types of births for all 366 dates. No reply yet, unfortunately. (Fair enough; it’s a 15-year-old paper, and the authors may well have lost the data file.)

There was some discussion in comments on the effects of scheduled births on the pattern of dates of all three sorts of births. It’s tricky because whether a birth is scheduled or not is itself “endogenous” in that a birth could be scheduled but then the baby could be born before that date.

More data!

I wasn’t sure about what to say here in the absence of more data . . . and then some more data showed up! I did some searching and came across this article by Mireille Jacobson, Maria Kogelnik and Heather Royer on birth timing and post-natal outcomes, who write:

Fewer births occur on major US holidays than would otherwise be expected. We use California data to study the nature and health implications of this birth date manipulation. . . . “missing” holiday births are displaced to a window of time 11 days before the holiday through 16 days after the holiday. Delivery type does not change over this window, consistent with a pure retiming of births rather than an increase in the use of procedures such as cesarean sections. . . . while some of the retiming seems to be driven by patients’ preferences, provider incentives appear to play a crucial role in holiday-related birth retiming. At Kaiser Permanente hospitals, where systemwide financial incentives discourage providers from electively timing births, the dip in births on holidays is less than for hospitals overall.

Here’s what they estimate:

No plots of the raw data, unfortunately just this estimate which is based on averaging over a set of holidays that occur throughout the year.

Here’s where the data come from:

Our primary data source is the restricted-access 2000-2016 California Birth Statistical Master Files. These data cover the universe of California births during this period and come from birth certificate information that the parents and medical provider fill out at the time of birth. These data include demographic information (e.g., age, education) for the parents, health conditions/outcomes of the mother and infant (e.g., gestational diabetes, birth weight, gestational length), and the use of medical interventions (e.g., cesarean section, induction, and stimulation). Crucial to our approach, these data include the exact date of birth of the infant.

And here’s a quick summary:

Births average 1442 per day, but are systematically lower on holidays and weekends (with a mean of about 1100 births per day) than on other days. The data on delivery mode make clear that this is a result of scheduling. The number of cesarean section deliveries is nearly 50% lower on holidays and weekends than on other days. Induced/stimulated births are about 28% lower. Spontaneous vaginal births are also lower on holidays (by about 15%), although they account for a much higher share of births on holidays (52%) than on other non-weekend days (44%).

That’s cool. Now I want to see the raw data. It says they’re in the California Birth Statistical Master Files. I don’t know where to find these files, also if they’re “restricted access,” maybe I don’t have permission to see them, and I guess that the authors of the above-linked paper won’t be able to send them to me.

A request

If anyone out there has the data from the California Birth Statistical Master Files and wants to plot the time series of avg #births by date (and multiply the number for 29 Feb by (# years in data)/(# leap years in data)), for each of the three categories of births, could you please graph these (three time series on a single plot would be fine) and send to me? Thanks!

“The difference between ‘significant’ and ‘not significant’ is not itself statistically significant” . . . in the wild!

Here’s the statistical principle (from my 2006 paper with Hal Stern).

And here’s the recent example:

Kevin Lewis points us to this article, “The long-term effects of marijuana use on mental health outcomes,” which begins:

We estimate the long-term effect of initiating marijuana use in adolescence on several mental health outcomes later in life. We use the first two waves (1994–1996) and the fifth wave (2016–2018) of the National Longitudinal Study of Adolescent to Adult Health (Add Health) and estimate instrumental variables models with school-level fixed effects, where the instrument is the respondents’ perceptions about their friends’ marijuana use. We find that marijuana use in adolescence is associated with a significant increase in anxiety approximately twenty years later. The increase in anxiety is only present among females and is stronger among females who used marijuana regularly as adolescents.

Did you catch the error?

Read that last sentence carefully: the phrase “only present among females” has the flavor of comparison of significance levels.

And, indeed, here are the results:

You can just look at the rightmost column (“Anxiety”). The estimate is 0.05 +/- 0.04 for boys and 0.11 +/- 0.05 for girls. A quick calculation gives an approximate estimated difference of 0.06 +/- 0.06: not statistically significant!

I’m also suspicious of the “stronger among females who used marijuana regularly”—this is based on a coefficient estimate getting larger when they modify their marijuana use indicator, and again this is not the sort of direct comparison that would be recommended.

It’s not that I’m saying that non-statistically-significant results should be ignored. The authors should report all comparisons of interest. It’s just that comparisons should come with uncertainties. Comparing significance levels doesn’t do that. They’re creating a misleading sense of certainty in their analysis. I have no reason to think they’re doing it wrong on purpose; it’s just an unfortunate side effect of dichotomizing results based on significance thresholds.

The point of this post is not to focus on this particular article published in an obscure journal. It’s just yet another example of something we’ve seen many times before and unfortunately will see many times again.

Plagiarist USC celebrity doctor: What’s he been up to lately?

Is this an April Fool’s post? I’m not sure.

David Agus is the author of a book that included “at least 95 separate passages” of plagiarism, including my personal favorite (as reported by the LA Times): “Long sections of a chapter on the cardiac health of giraffes appear to have been lifted from a 2016 blog post on the website of a South African safari company titled, ‘The Ten Craziest Facts You Should Know About A Giraffe.'”

It’s hard to know where to start with this. Maybe the first thing to note is that Agus is an oncologist and has no expertise on the cardiac health of giraffes. Who writes a book with an entire chapter on something he knows nothing about? Oh yeah, there were these guys . . . but at least when they were writing about something they knew nothing about, they wrote it in their own words.

I was particularly annoyed that Agus wrote, “I’m not pitching a tent to watch chimpanzees in Tanzania or digging through ant colonies to find the long-lived queen, for example . . . I went out and spoke to the amazing scientists around the world who do these kinds of experiments, and what I uncovered was astonishing.” Ummm, maybe this is the case with the ant colonies, but it obviously isn’t the case with the giraffes–I’m still stunned that he did an entire chapter on them! He didn’t speak with any scientists, “amazing” or otherwise. He just ripped off the website of a safari company.

I’d say, “What a loser,” but in the usual scheme of things this guy’s a winner. He still appears to be employed by USC–he’s the “founding director and CEO of the Lawrence J. Ellison Institute for Transformative Medicine”–despite clearly violating their ethics code:

But that was a couple years ago. What’s the good doctor been up to lately?

18 Sep 2024: He’s featured in the Princeton alumni magazine, described as a “cancer specialist, researcher, and wellness advocate.” They forgot “author of a plagiarized book on giraffes”! Also this: “Sometimes he reminds patients that there is no miraculous cure-all.” I don’t know: plagiarism is a miraculous cure-all to writer’s block, is it not?

22 Oct 2024: He’s featured in Time Magazine: “‘Interventions that can affect outcome are what we need, and we need companies to put the capital up front and do the studies to show they can affect outcome,’ says Agus. ‘And once we do that, those are the technologies that we should all push and enable our patients to use, and they have to be accessible.'” And we should believe this, because . . . why, exactly? Because a rich USC med school professor with a demonstrated willingness to lie has money on the line? To be fair, the Time article does identify Agus as “founding director and co-CEO of the Ellison Institute of Technology.” But then I’d hope they’d take the next step and express a little skepticism about his motivations.

23 Feb 2025: TechCrunch reports: “Oracle co-founder [Larry Ellison] set out to reinvent agriculture on Hawaii’s Lāna’i Island, which he scooped up for $300 million back in 2012. . . . Ellison dreamed of AI-powered greenhouses and robot harvesters feeding the world sustainably. Instead, Sensei has been tripped up by tech snarls . . . and rookie mistakes. . . . Sensei, co-founded by a medical doctor [yup, David Agus, head of the “Lawrence J. Ellison Institute for Transformative Medicine”] and led currently by a tech exec who runs Sensei from Boston, has had small wins, reports the WSJ. Its lettuce and cherry tomatoes now appear at the island’s few local markets and restaurants. But constant delays, leadership shake-ups, and pricey blunders, including cannabis grow houses that needed to be gutted and rebuilt, highlight a tough truth: even bottomless funding is no match for the hard lessons of a specialized industry.”

Hmmm . . . I think the real lesson here is that copy-and-pasting material from a website on giraffes does not qualify you to run an agriculture company.

This was all a year after “Ellison-backed med tech startup Project Ronin closes doors . . . founded by Dave Hodgson, a tech leader with a history in medicine; David Agus, the cancer expert who treated Steve Jobs . . .; serial health entrepreneur Rowan Chapman; and Larry Ellison, Oracle founder and CTO.” Agus also treated Madonna, so, sure, why not throw a few million dollars at the guy.

What’s funny (actually not funny) to me is that people like Agus who are known liars still get trusted with money and resources in this way. The people in charge must think that Agus only lies to other people, never to them. Or maybe Agus is just a figurehead in these schemes: perhaps Ellison and USC are paying for his celebrity name, in which case they have a motivation to minimize his ethical transgressions.

So, to return to the first line of this post: Are USC and Larry Ellison fools for paying this guy? Are Princeton University, Time Magazine, Los Angeles magazine, and the New Yorker magazine fools for repeatedly promoting him? Or is this all transactional: they puff up his celebrity, and then they get to bask in the celebrity connection?

The reciprocal betrayals of Saul Bellow and Paul Meehl

Saul Bellow was a critically-acclaimed, best-selling author. Paul Meehl was an obscure psychology professor with a sideline in the philosophy of science.

That said, to the audience of this blog, Meehl may be more famous than Bellow. At least, we may spend more time thinking about Meehl’s critiques of science than of Bellow’s characters, and we might even spend more time reading Meehl (for example, here and here) than Bellow.

But here’s something you might not know: Meehl was Saul Bellow’s therapist! From Bellow biographer James Atlas:

Bellow saw four psychiatrists during his lifetime: Dr. Chester Raphael, a Reichian who practiced in Queens and who was the model for Dr. Sapir in his unfinished novel about Rosenfeld; Paul Meehl, a psychologist in Minneapolis he had consulted during the disintegration of his second marriage, when he was teaching at the University of Minnesota, Albert Ellis, the famous “sexologist” whom Bellow saw for what he once described as “pool room work,” or sexual technique; and Heinz Kohut.

The meetings with Meehl came when Bellow was a visiting professor at the University of Minnesota. Atlas continues:

I [Atlas] had interviewed the first three, all of whom were willing, no doubt out of vanity, to violate patient/doctor (or psychologist) confidentiality.

This made me sad. The great Paul Meehl, violating therapist-patient confidentiality!

Then again, Atlas also writes that Meehl was the model for Dr. Edwig in Bellow’s novel Herzog. I’ve never read that one, so I did a little bit of googling . . . it seems that Edwig is not a major character in the book, but I did come across one source that says that Edwig “is described as a dupe.”

So maybe Meehl, having felt betrayed by Bellow for how he was described in the novel, thought it was ok to betray Bellow in turn. Or maybe he had the opinion that Bellow, as a public figure, did not deserve the usual sort of patient confidentiality. Or maybe doctor-patient confidentiality wasn’t such a thing back in the 1950s. I don’t know. This short memoir by Meehl discusses his motivations for becoming a therapist, but Saul Bellow never comes up.

Anyway, I remain a fan of Meehl–I’m not saying we should “cancel” the guy–; it’s just an interesting episode, an unexpected historical encounter between two worlds.

Conflicts of interest in vaping studies. (Yes, researchers will risk their professional reputations out of some combination of political motivations, irritation, and money.)

Paul Alper points us to this news report that states:

Two New York University (NYU) professors were recently found to have collaborated directly with executives from the vaping company Juul without disclosing these relationships to academic journals or Congress. This revelation came to light during a STAT investigation. At a time when the youth vaping crisis was at its peak, these professors, David Abrams and Ray Niaura, emerged as authoritative voices defending vaping as an effective public health strategy for adults to cut back or quit smoking, despite its growing popularity among youth.

David Abrams, a frequent commentator on vaping in the news media, coordinated extensively with Juul on public messaging in 2017 and 2018. He asked Juul officials for talking points, allowed company executives to review an academic article before publishing, and attended Juul scientific advisory board meetings—all without disclosing these connections to journal publishers or the public.

Ray Niaura was also involved in collaborating with Juul executives but did not disclose these relationships.

Alper asks:

Do some people have a secret death wish? Do Abrams and Niaura not realize that their public utterances will be scrutinized? Yet once again, all that fuss over Francesca Gino’s supposed manipulations do not compare with what harm is being done in other fields involving cancer and cancer causing products.

I have two responses.

1. Researchers have seriously damaged their reputation by working for cigarette companies, even without any suggestion of research misconduct or hidden conflicts or interest. Two prominent examples are R. A. Fisher and Donald Rubin, both of whom have been hugely influential in statistics and many applied fields, and remain very respected, but still come off looking pretty foolish for their notorious edgelord positions on smoking and health (for example, here’s Fisher referring to anti-smoking campaigns as “terrorist propaganda,” and I personally heard Rubin argue that smoking is not addictive). For some reason, prominent statistician Ingram Olkin isn’t so notorious in this regard, perhaps because he just took cigarette money and kept quiet about it, without than taking any strong stances on the issue.

In regard to Alper’s questions, I don’t think Fisher, Rubin, or Olkin had any professional “death wishes”; I’m guessing they were motivated by some combination of political views, irritation at the statistical weaknesses of some anti-smoking arguments, and money. And, hey, these are strong motivations for me too! Not regarding smoking, but on other statistical and policy issues I’ve written about.

2. Regarding the specifics of the case, this news is not surprising. The names David Abrams and Ray Niaura rang a bell. From my post in Feb 2020 on the topic:

I looked up David Abrams, the first author of the letter sent to the journal. He’s a professor of public health at New York University, his academic training is in clinical psychology, and he’s an expert on cigarette use. For example, one of his recent papers is, “How do we determine the impact of e-cigarettes on cigarette smoking cessation or reduction? Review and recommendations for answering the research question with scientific rigor,” and another is “Managing nicotine without smoke to save lives now: Evidence for harm minimization.” A web search brought me to this article, “Don’t Block Smokers From Becoming Smoke-Free by Banning Flavored Vapes,” on a website called Filter, which states, “Our mission is to advocate through journalism for rational and compassionate approaches to drug use, drug policy and human rights.” Filter is owned and operated by The Influence Foundation, which has received support from several organizations, including Juul Labs, Philip Morris International, and Reynolds American, Inc. . . .

So, several of the people involved in this controversy have conflicts. In their letter to the journal, Abrams et al. write, “The signatories write in a personal capacity and declare no competing interests with respect to tobacco or e-cigarette industries.” I assume this implies that Abrams is not directly funded by Juul Labs, Philip Morris International, etc.; he just writes for an organization that has this funding, so it’s not a direct competing interest. But in any case these researchers all have strong pre-existing pro- or anti-vaping commitments.

Hmmm . . . so maybe this was a competing interest! If Abrams and Niaura were asking Juul officials for talking points, allowed company executives to review an academic article before publishing, and attended Juul scientific advisory board meetings . . . it does sound like they were expecting compensation from Juul in some form or another, right? I can’t really say. The linked news article is paywalled so I don’t know what is in these documents that they cite.

In any case, if the researchers really “attended Juul scientific advisory board meetings,” then, yeah, that sounds like a conflict of interest to me. Much more of a conflict than simply publishing an article on a website funded by Juul.

“Why are places like Stanford and Johns Hopkins hosting gatherings of well-known coronavirus cranks?”

Joshua in comments asks:

Why are places like Stanford and Johns Hopkins hosting gatherings of well-known coronavirus cranks?

My reply:

Considering that about half the electorate plans to vote for Donald Trump, who endorses lots of conspiracy theories that are much more ridiculous than those of the coronavirus cranks–and lots of non-Trump voters also believe in conspiracy theories of their own–it makes sense that some nonzero percentages of Stanford and Johns Hopkins faculty would support such views and some additional percentage would consider it a savvy political play to endorse these views. On the other hand, yeah, there are lots of minority views that still aren’t supported by conferences at these universities. For example, I’d guess that more than 10% of Stanford and Johns Hopkins faculty believe in astrology, but they’re not about to host conferences on the topic, as it would be just too embarrassing.

I think that if the coronavirus cranks did not have major political support, they wouldn’t be holding these conferences.

The other thing is that a lot of people are rightfully annoyed by the b.s. coming from mainstream coronavirus authorities, so they feel like it’s only fair to give the coronavirus cranks a hearing, as, for all their flaws, they’re providing an alternative perspective.

P.S. Just to be clear I don’t think that anyone is claiming that all the speakers at these conferences are coronavirus cranks. Many of them, though, yeah, they are. Remember these people?

P.P.S. Oddly enough, I spoke at Stanford the day before that covid conference—and I was talking about the political content of unreplicable research, in a seminar series whose previous speakers included some of the speakers at the covid conference. It’s a close-knit world.

“Florida man eats diet of butter, cheese, beef; cholesterol oozes from his body”: How much am I to blame for this?

Oh, this is horrible, no joke:

What could go wrong with eating an extremely high-fat diet of beef, cheese, and sticks of butter? Well, for one thing, your cholesterol levels could reach such stratospheric levels that lipids start oozing from your blood vessels, forming yellowish nodules on your skin.

That was the disturbing case of a man in Florida who showed up at a Tampa hospital with a three-week history of painless, yellow eruptions on the palms of his hands, soles of his feet, and elbows. His case was published today in JAMA Cardiology.

The man, said to be in his 40s, told doctors that he had adopted a “carnivore diet” eight months prior. His diet included between 6 lbs and 9 lbs of cheese, sticks of butter, and daily hamburgers that had additional fat incorporated into them. Since taking on this brow-raising food plan, he claimed his weight dropped, his energy levels increased, and his “mental clarity” improved.

I’m uncomfortably reminded of my friend Seth Roberts, who died of heart failure not long after adopting a diet in which he ate half a stick of butter a day, which he claimed from self-experimentation to have improved his brain function.

The Florida man was ingesting a lot more than half a stick of butter per day; still, I was struck by his report of improved mental clarity, which uncomfortably echoed Seth’s claims from a decade earlier.

I feel some very small responsibility for this chain of events. Seth was a friend of mine at the University of California in the early 1990s–we taught a couple of courses together–and he kept me informed of his work on self-experimentation. In 2005 he published a research article on the topic, and I promoted it on the blog. Even then, his self-experimentation was unusual, but he hadn’t gone off the deep end. We had further blog discussion that year (also here), and Alex Tabarrok linked to it from his Marginal Revolution blog. From there it was picked up by Stephen Dubner and Steven Levitt in the New York Times. At the time I thought this multiple stage of amplification was pretty cool, and at this point Seth didn’t need my help to get publicity. Dubner and Levitt invited him to guest-blog on his diet at Freakonomics, then a year later Seth published a successful diet book, which I reviewed positively, as did Tabarrok. Around that time, I also published a couple of conversations with Seth on his research; see here and here. Seth also started his own blog and website where people using his diet could share tips and otherwise communicate. He was followed by mid-level celebrities such as Dennis Prager and Tucker Max.

Seth was an early figure in the paleo-lifestyle movement. Indeed, he was telling me his theories about the benefits of caveman diet and caveman lifestyle many years before I heard it anywhere else. In the late 1990s or early 2000s he even wrote a book on the topic, but he never found a publisher, and I guess he abandoned the project after publishing his diet book.

Going through the blog archives, I was surprised to learn that, as early as 2007, Seth was claiming cognitive benefits from oil consumption. He shared data showing it had improved his reaction time. I think this must have been a byproduct of him drinking oil as part of his weight-loss plan. Here’s what I wrote back in 2007:

Encouraged by the success of his self-experimentation to help his sleep, mood, and weight concerns, Seth Roberts has been experimenting with the effects of drinking flaxseed oil. . . . Commenting on another recent one of Seth’s self-experiments, I wrote,

Seth,
Not to be a wet blanket or anything, but aren’t you worried that your findings might be due to expectation effects: you knew which oil you were taking when doing the tests, right?

Seth replied,

Andrew, no, I’m not worried that the results are due to expectations. If the results always conformed to my expectations, I’d be worried, but they haven’t — see my post about eggs. Moreover, this particular result confirms a result that was a surprise. In other words, I’ve gotten the same result when I was expecting it and when I wasn’t expecting it.

I’m still concerned, though. Seth is saying that it’s not just an expectation effect because he wasn’t always expecting the results. But I could see a bias arising from positive feedback, as follows: You try a new treatment and then see what happens after, with no expectations except that things might change. There is some noise to this measurement–just at random, it will be higher or lower than before. Having seen this, you adjust your expectations; this then affects your next measurement, etc.

In retrospect, I think I was too mild. I suggested to Seth: “maybe you could get a partner in experimentation, someone who lives or works nearby, and he or she could give you a randomly assigned oil. That is, your partner would know which oil you’re getting, but you wouldn’t. In fact, you wouldn’t even know if you were being given something new that day.” Seth agreed but wrote, “they are low on my to-do list.”

Again, I wish I’d pushed back harder on that:

– A clean negative result could’ve given Seth (and me!) a healthy dose of skepticism and motivated him to think more carefully about his theories when deciding what experiments to try next.

– A clean positive result could’ve given him and his followers a level of confidence and motivated him to experiment more systematically, rather than spiraling out of control with goofy and possibly dangerous ideas such as eating a half a stick of butter a day.

We had another exchange in 2009, where, again, I said, “maybe this could all be a confirmation bias,” and, again, Seth pushed back, and, again, I now wish I’d taken a firmer line.

As the years went on, Seth gained confidence and his experiments became more and more dubious, culminating in the stick-of-butter thing which, maybe it wasn’t fatal, but maybe it was, and I doubt it was helpful, either to his well-being or others’.

This is a long chain of events, so I don’t feel a high degree of responsibility for my small part in encouraging the crazier aspects of the paleo lifestyle movement. But, to the extent that I was proud that my promotion of Seth’s work led to his media success, I should correspondingly feel bad about the negative consequences.

What do I think of this Bayesian analysis of the origins of covid?

A colleague links to this recent article, “A Bayesian Assessment of the Origins of COVID-19 using Spatiotemporal and Zoonotic Data,” written by economist Andrew Levin, and asks, “What do you think of this analysis? Is it sound? Does it mean a sensible person should be very convinced it was a lab leak?”

Here’s the abstract of the paper in question:

This paper uses Bayesian methods in conjunction with spatiotemporal and zoonotic data to evaluate the odds ratio for two hypotheses regarding the origin of the COVID-19 pandemic, namely, an accidental laboratory leak of a chimera virus or the transmission of a natural virus from an infected wildlife mammal. The overall Bayes factor is decomposed into 4 components: (1) the odds that the outbreak would occur in the People’s Republic of China (PRC); (2) the odds that the outbreak would occur in Wuhan, conditional on its location in PRC; (3) the odds of observing the spatiotemporal pattern of confirmed COVID-19 cases with no known link to the specific wholesale market where wildlife mammals were being sold, conditional on the outbreak taking place in Wuhan; and (4) the odds of observing the spatiotemporal pattern of confirmed vendor cases at that market, conditional on the outbreak taking place in Wuhan. These four conditional Bayes factors are estimated as 2.3:1, 20:1, 27:1, and 12:1, respectively, and hence the overall odds ratio is 14,900:1, indicating overwhelming evidence in favor of the hypothesis that the pandemic resulted from an accidental lab leak. This conclusion is robust to alternative specifications of the detailed statistical analysis.

My answer to my colleague’s question posed above is, I don’t know.
I corresponded with the author of that paper as well as the author of the paper linked here. It’s hard to compute Bayesian probabilities for this problem, not so much because of the priors but because of the likelihood, which is the probability of the data given the model. One problem is the selection of what is considered to be data, the other problem is that the model (“lab leak” or “wet-market leak” or whatever) is not clearly specified–in statistics jargon, these are “composite hypotheses.” This is not a criticism of this particular paper per se; it’s just a general difficulty with this sort of analysis. It’s not clear to me that Bayesian inference is the right way to attack this sort of problem. But I’ve been intimidated by the technical biological details in all these analyses so I haven’t looked at them personally.