Skip to content
Archive of posts filed under the Causal Inference category.

No, I don’t believe that claim based on regression discontinuity analysis that . . .

tl;dr. See point 4 below. Despite the p-less-than-0.05 statistical significance of the discontinuity in the above graph, no, I do not believe that losing a close election causes U.S. governors to die 5-10 years longer, as was claimed in this recently published article. Or, to put it another way: Despite the p-less-than-0.05 statistical significance of […]

The value of thinking about varying treatment effects: coronavirus example

Yesterday we discussed difficulties with the concept of average treatment effect. Part of designing a study is accounting for uncertainty in effect sizes. Unfortunately there is a tradition in clinical trials of making optimistic assumptions in order to claim high power. Here is an example that came up in March, 2020. A doctor was designing […]

Understanding the “average treatment effect” number

In statistics and econometrics there’s lots of talk about the average treatment effect. I’ve often been skeptical of the focus on the average treatment effect, for the simple reason that, if you’re talking about an average effect, then you’re recognizing the possibility of variation; and if there’s important variation (enough so that we’re talking about […]

The point here is not the face masks; it’s the impossibility of assumption-free causal inference when the different treatments are entangled in this way.

Adam Pearce writers: When I read your Another Regression Discontinuity Disaster post last year, I was curious how much shifting the breakpoint would change the fit lines. A covid paper making the rounds this weekend used a similar technique so I hooked it up to an interactive widget that lets you tweak the start and […]

Challenges to the Reproducibility of Machine Learning Models in Health Care; also a brief discussion about not overrating randomized clinical trials

Mark Tuttle pointed me to this article by Andrew Beam, Arjun Manrai, and Marzyeh Ghassemi, Challenges to the Reproducibility of Machine Learning Models in Health Care, which appeared in the Journal of the American Medical Association. Beam et al. write: Reproducibility has been an important and intensely debated topic in science and medicine for the […]

How should those Lancet/Surgisphere/Harvard data have been analyzed?

As you will recall, the original criticism of the recent Lancet/Surgisphere/Harvard paper on hydro-oxy-whatever was not that the data came from a Theranos-like company that employs more adult-content models than statisticians, but rather that the data, being observational, required some adjustment to yield strong causal conclusions—and the causal adjustment reported in that article did not […]

Sequential Bayesian Designs for Rapid Learning in COVID-19 Clinical Trials

This from Frank Harrell looks important: This trial will adopt a Bayesian framework. Continuous learning from data and computation of probabilities that are directly applicable to decision making in the face of uncertainty are hallmarks of the Bayesian approach. Bayesian sequential designs are the simplest of flexible designs, and continuous learning capitalizes on their efficiency, […]

This one’s for the Lancet editorial board: A trolley problem for our times (involving a plate of delicious cookies and a steaming pile of poop)

A trolley problem for our times OK, I couldn’t quite frame this one as a trolley problem—maybe those of you who are more philosophically adept than I am can do this—so I set it up as a cookie problem? Here it is: Suppose someone was to knock on your office door and use some mix […]

“The good news about this episode is that it’s kinda shut up those people who were criticizing that Stanford antibody study because it was an un-peer-reviewed preprint. . . .” and a P.P.P.S. with Paul Alper’s line about the dead horse

People keep emailing me about this recently published paper, but I already said I’m not going to write about it. So I’ll mask the details. Philippe Lemoine writes: So far it seems you haven’t taken a close look at the paper yourself and I’m hoping that you will, because I’m curious to know what you […]

This is not a post about remdesivir.

Someone pointed me to this post by a doctor named Daniel Hopkins on a site called KevinMD.com, expressing skepticism about a new study of remdesivir. I guess some work has been done following up on that trial on 18 monkeys. From the KevinMD post: On April 29th Anthony Fauci announced the National Institute of Allergy […]

Alexey Guzey’s sleep deprivation self-experiment

Alexey “Matthew Walker’s ‘Why We Sleep’ Is Riddled with Scientific and Factual Errors” Guzey writes: I [Guzey] recently finished my 14-day sleep deprivation self experiment and I ended up analyzing the data I have only in the standard p < 0.05 way and then interpreting it by writing explicitly about how much I believe I […]

Be careful when estimating years of life lost: quick-and-dirty estimates of attributable risk are, well, quick and dirty.

Peter Morfeld writes: Global burden of disease (GBD) studies and environmental burden of disease (EBD) studies are supported by hundreds of scientifically well-respected co-authors, are published in high level journals, are cited world wide and have a large impact on health institutions‘ reports and related political discussions. The main metrics used to calculate the impact […]

Doubts about that article claiming that hydroxychloroquine/chloroquine is killing people

James Watson (no, not the one who said that cancer would be cured by 2000, and not this guy either) writes: You may have seen the paper that came out on Friday in the Lancet on hydroxychloroquine/chloroquine in COVID19 hospitalised patients. It’s got quite a lot of media attention already. This is a retrospective study […]

New report on coronavirus trends: “the epidemic is not under control in much of the US . . . factors modulating transmission such as rapid testing, contact tracing and behavioural precautions are crucial to offset the rise of transmission associated with loosening of social distancing . . .”

Juliette Unwin et al. write: We model the epidemics in the US at the state-level, using publicly available death data within a Bayesian hierarchical semi-mechanistic framework. For each state, we estimate the time-varying reproduction number (the average number of secondary infections caused by an infected person), the number of individuals that have been infected and […]

This one’s important: Designing clinical trials for coronavirus treatments and vaccines

I’ve had various thoughts regarding clinical trials for coronavirus treatments and vaccines, and then I came across thoughtful posts by Thomas Lumley and Joseph Delaney on vaccines. So let’s talk, first about treatments, then about vaccines. Clinical trials for treatments The first thing I want to say is that designing clinical trials is not just […]

If the outbreak ended, does that mean the interventions worked? (Jon Zelner talk tomorrow)

Jon Zelner speaks tomorrow (Thurs) at 1pm: PREDICTING COVID-19 TRANSMISSION In this talk Dr. Zelner will discuss some ongoing modeling work focused on understanding when we can and cannot infer that interventions meant to stop or slow infectious disease transmission have actually worked, and when observed outcomes cannot be distinguished from selection bias. Dude’s an […]

“Positive Claims get Publicity, Refutations do Not: Evidence from the 2020 Flu”

Part 1 Andrew Lilley, Gianluca Rinaldi, and Matthew Lilley write: You might be familiar with a recent paper by Correira, Luck, and Verner who argued that cities that enacted non-pharmaceutical interventions earlier / for longer during the Spanish Flu of 1918 had higher subsequent economic growth. The paper has had extensive media coverage – e.g. […]

Imperial College report on Italy is now up

See here. Please share your reactions and suggestions in comments. I’ll be talking with Seth Flaxman tomorrow, and we’d appreciate all your criticisms and suggestions. All this is important not just for Italy but for making sensible models to inform policy all over the world, including here.

NPR’s gonna NPR (special coronavirus junk science edition)

1. The news! Zad’s cat, pictured above, is not impressed by this bit of cargo-cult science that two people sent to me: No vaccine or effective treatment has yet been found for people suffering from COVID-19. Under the circumstances, a physician in Kansas City wonders whether prayer might make a difference, and he has launched […]

10 on corona

Here are some things people have sent me lately. They are in no particular order, except that I put the last item last so we could end with some humor. After this, I’ll write a few more blog posts, then it’ll be time to do some real work. Table of contents 1. Suspicious coronavirus numbers […]