Category Archives: Causal Inference

Survey Statistics: Times/Siena Polls and Times Voter Trends

Posted on by
Reply

Yesterday Andrew wrote about Alan Zaslavsky, who recently passed.

This is typical of his work in that it was innovative, with applied importance, and understated.

As Fan Li quoted, Alan was “the best applied statistician on earth”. I’ll write more Alan stories soon. He was my PhD advisor and hero.

Since it is election day, let’s talk about methods to analyze polls, where Alan’s legacy can’t be overstated. Fan Li mentioned work with Alan on balancing covariates, cited by Huling & Mak (2024) in their work on energy balancing weights, which are used by the Times/Siena Poll. This Survey Statistics series has discussed a few Times/Siena Poll decisions over the last year.

This week, in “The Final Polls Show a Blue Surge. Can We Trust Them?” Nate Cohn compares how the 2026 polls would look without these big changes, and sees they would be on average more than 3 points higher for Democrats.

Last week Andrew wrote about MRP at the NYT. So cool ! Their methodology article says:

The estimates do not report a margin of sampling error because they are not based on a sample of the population. The estimates are derived from multiple models, which are each subject to different kinds of error.

Yes, there are many kinds of error to consider, in “Margin of Error” we discussed how to get an MRP margin of error assuming no nonresponse bias after the MRP adjustments, no measurement error, and no model misspecification. In “Total Margin of Error” we saw that in election polls from 1998 to 2014, Shirani-Mehr et al. found that total survey error is about twice the reported margins of error. Even if you have margins of error to report, visualizing them is challenging (see Gelman and Price 1999 “All maps of parameter estimates are misleading”).

I was curious about the relationship between their MRP methods and their weighting methods. (See “MRPW”, “struggles with equivalent weights”, continued struggles, and “equivalent models, equivalent weights (locally)”, and “structured MRP to smooth survey weights”.) The only mention I found of their weights is:

The polls were joined to the given year’s voter file and reweighted to match the voter file by L2’s modeled race, party, age and interactions thereof, with the original survey weight as the base weight. The adjusted survey weight was used in all subsequent race models.

So they calibrate the base weights to match the voter file, see Lumley 2010 Chapter 7.

It’s been a few months since I’ve blogged about the logit shift, so it is time. (See: “2 flavors of calibration”, 2nd helpings (multinomial outcomes), 3rd helpings (multivariate outcomes), 4th helpings (more on multivariate outcomes), “logit shift and raking”.) The NYT uses a version of this shift:

The resulting individual-level probabilities were adjusted in multiple steps to match the actual election result by precinct. First, the probability that a voter would support a minor party candidate was adjusted to match the minor party proportion of the vote. Second, the probability that a voter would support a major party candidate was adjusted to match a candidate’s major party vote share in the precinct. Finally, the probability that a voter would support the Democratic or Republican candidate was calculated by multiplying the adjusted Democratic and Republican share of the major party vote by the adjusted major party vote share of the precinct.

Presumably they’re doing something like:

  1. E(shift_a(P(major party | X))) = P(major party) solve for a.
  2. Then either:
    1. E(shift_b(P(Dem | major party, X))) = P(Dem | major party) solve for b. Or
    2. E(shift_a(P(major party | X)) * shift_b(P(Dem | major party, X))) = P(Dem) solve for b.

2A isn’t quite right: it implies E(shift_a(P(major party | X))) * E(shift_b(P(Dem | major party, X))) = P(Dem). But we want 2B to hold, and the two are equivalent only if the individual-level probabilities for major party and for Dem given major party are uncorrelated.

It’s hard to write prose to describe something technical. The NYT level of methods transparency is great !

Problems with the “Do no harm” principle in the presence of uncertainty

Posted on by
Reply

In a paper entitled, Counterfactual Harm: A Counter-argument, Amit Sawant and Mats Stensrud write:

A core principle in medicine is non-maleficence, often equated with “do no harm.” A formal definition of harm based on counterfactual reasoning has been proposed and popularized. This notion of harm has been promoted in simple settings with binary treatments and outcomes. Here, we highlight a problem with this definition in settings involving multiple treatment options. Illustrated by an example with three tuberculosis treatments (say, A, B, and C), we demonstrate that the counterfactual definition of harm can produce intransitive results: B is less harmful than A, C is less harmful than B, yet C is more harmful than A when compared pairwise. . . . In contrast, an interventionist definition of harm based on expected utility forgoes counterfactual comparisons and ensures transitive treatment rankings.

I agree, and this is related to what we wrote in our Russian roulette paper. I’ve always been uncomfortable with utility functions that depend on outcomes that have never occurred. I recognize that counterfactual utilities can make sense psychologically (this includes concepts such as uncertainty aversion and regret), but not normatively, not for what I’ve referred to as institutional decision analysis. This new paper by Sawant and Stensrud goes beyond our Russian roulette paper by introducing a new example and considering continuous outcomes.

Also related is this paper by Stensrud with Aaron Sarvet, “Rejoinder to ‘”Harm” in personalized medicine–an alternative perspective.'”

Stochastic potential outcomes: A discussion with Ang Yu and Jonas Mikhaeil

Posted on by
Reply

Ang Yu read the Russian Roulette paper (by Jonas Mikhaeil and me) and wrote:

I was wondering whether it might be useful to alternatively frame the problem in terms of stochastic versus deterministic interventions as opposed to potential outcomes. I have attached a short note on your paper and would appreciate your thoughts.

Jonas replied:

1. If I [Jonas] understand correctly, the interventional framework is meant to avoid counterfactuals altogether (e.g. Didelez, et al., 2012), right? In this case, one might be able to avoid the problems with “counterfactual decision problems” altogether by rephrasing them as classical decision problems under interventions?
Maybe this is exactly what you have in mind. I think this would be a very interesting conceptual contribution.

2. What I like about the stochastic potential outcomes perspective is that it allows us to model which uncertainty/variation we want to incorporate in the definition of the treatment (bullet vs. pistol). By defining the utility as a functional over the distribution of the stochastic potential outcome, we can integrate out the uncertainty that remains after treatment assignment. With this additional model choice, we can choose which uncertainty is irrelevant (and hence integrated out prior) to our decision problem.

3. I’m not entirely sure I understand how this would be accomplished by switching from deterministic to stochastic interventions. I’m sure this is an oversight on my part, and I would be excited to see this perspective be fleshed out.
My immediate question would be the following: potential outcomes are defined on the level of the individual, so it makes sense to distinguish between averaging over the treatment vs. averaging over an individual’s outcome. Accordingly, we can define the utility by only averaging over an individual’s outcome, fixing a treatment, and then consider the expected utility by averaging over treatment assignment.
In the interventional framework, I’m not sure how one would distinguish between the outcome Y of an individual being deterministic vs. stochastic for a given intervention? Because of this, I’m not clear how one would first integrate out individual-level uncertainty to define the utility and then calculate the expected utility based on treatment assignment in the interventional framework. (i.e. can what you call U(R6) be understood as an average over an individual’s outcome given a treatment?) I’m sure this is my oversight. I would love to hear what you think.

4. Following up on point 3: Does the Russian roulette paradox appear/disappear when switching from a deterministic vs. stochastic interventions perspective? Essentially, I’m wondering if the choice of how to model the intervention (deterministic vs. stochastic) allows to include/integrate out different levels of uncertainty in the decision problem.

Ang responded:

1. I think so. Didelez et al. 2012 was one of the earliest works in the stochastic intervention literature. Terminology and notation evolved over time in the literature, but their “regime” is the same as what I call “intervention”, and their distinction between atomic and random interventions is the same as my distinction between deterministic and stochastic interventions.

2. I agree that being able to integrate out sort of the “inner” uncertainty is important. If I understand your point correctly, my sense is simply that we already have an existing framework that can do exactly that: we just need to rephrase your point. In the stochastic intervention framework, intervention is pistol and treatment is bullet. So we can integrate over treatments and get the utility of an intervention for each individual.

3. I think the content of the stochastic intervention and stochastic potential outcomes frameworks is exactly the same, just different labels. In my mind, Y_{i, R6} for individual i is a random variable because of the randomness in the intervention R6. So we can call E[Y_{i, R6}] an average over the randomness of the potential outcomes or the randomness of the intervention. The difference is in naming, not substance. Does that make sense?

4. I think I’m not inventing a new way to solve to paradox, but just pointing out the connection between your solution and the literature on stochastic interventions.

My quick thought on all of this is that you can have stochasticism in the intervention or in the outcome conditional on the intervention. One way to see this is in the Russian roulette example: conditional on the chamber of the bullet, you can consider the intervention to be stochastic, but unconditional on the chamber, you can consider the outcome to be stochastic.

Or here’s an example with both sources of variation (or, you could say, “stochastic” aspects): Consider a treatment that is some sort of naturally growing herb that is gathered in the wild, so not a controlled compound. The intervention is stochastic because you might happen to get a sample of the herb that has a large amount of the active ingredient or you might get a dud that has very little. But the outcome will also be stochastic because the active ingredient will not always work. Actually, we could add a third level of variation by saying that (a) the active ingredient could work for some people and not others, and (b) even for the people for whom the ingredient works, it won’t always be effective.

For another example, consider physical therapy as a treatment for sciatica:

(i) To start with, there are lots of way that physical therapy is done; different physical therapists do different things. So the intervention of “physical therapy” is stochastic, if you don’t define ahead of time what will be done and who will be doing it.

(ii) For any given physical therapy intervention, there will be some people for whom it has no chance of working, because of the condition they have. This could be considered as stochastic, but I’d prefer to think of it as deterministic, conditional on a latent variable which contains details of the patient’s underlying medical condition.

(iii) If a patient is given a therapy that can work, the actual effectiveness will be stochastic. You could go full Rubin and consider the outcome as deterministic, conditional on a latent variable which contains the potential outcomes, but I think there can be a benefit to using a stochastic model, recognizing that there’s randomness in the course of the treatment.

For an even cleaner example, consider a coaching program to improve people’s basketball skills, where the outcome is a test where they take 100 shots and you count how many go in:

(i) Some coaches are better than others. So it makes sense to think of the intervention as stochastic.

(ii) Some people will be more amenable to coaching than others. For example, some people will be motivated by the coaching to practice; others won’t. I prefer to model this with a latent variable representing differences between people, but even then there can be a stochastic element: for example, maybe I’m the sort of person who would be motivated to practice, but I just happen to be very busy during the month of the coaching. You could code my state of busyness as a latent variable, but there could be all sorts of unmeasured or even unmeasurable factors affecting my receptiveness during that month, and at some point it makes more sense to bite the bullet and call it stochastic, in the same way that we consider a die roll to be stochastic even though in theory we could predict it pretty well given enough information on initial conditions.

(iii) Any particular coaching will work better for some people than others. There are different ways to have problems with your basketball shot, and different people will be able to improve in different ways. This could be considered as an interaction between the flavor of treatment and the player in the experiment, and it could be modeled with sufficient pre-treatment information on the players, but in practice it could make sense to model much or even all of this variation as stochastic.

(iiii) Finally, how many shots go in, conditional on the average probability of success? That’s stochastic. Again, you could follow Rubin (1974) and consider the outcome of the test to be deterministic, under the philosophy that “the great book of life has already been written; we just haven’t read it yet”; but to me it makes much more sense to consider the success probabilities or expected outcomes as fixed but unknown, with the actual outcomes as stochastic.

I’ve written before about the border between conditioning and randomness–it comes up in non-causal settings too. See here: https://sites.stat.columbia.edu/gelman/research/published/augie4.pdf and here: https://statmodeling.stat.columbia.edu/2022/12/23/a-probability-isnt-just-a-number-its-part-of-a-network-of-conditional-statements/

To which Ang responded:

I appreciate you laying out more generally different possible sources of randomness. On a philosophical level, they all make sense. From a practical perspective, though, I wonder if it’s useful to fully parse everything out. Stochastic interventions have proved useful in a lot of epidemiological and sociological applications. And I think the Russian roulette can also be framed as a stochastic intervention. In all those cases, it seems sufficient to explicitly consider only two sources of variation: the random variation in the intervention (e.g., bullet or no bullet), and the variation in potential outcomes across individuals. Please feel free to blog our discussion!

On a side note, I recently read VanderWeele and Hernan (2013), which framed a variant of stochastic intervention as a case of treatments with multiple versions. The stochastic intervention considered there also directly comes from a real empirical application.

I actually think we’re in agreement here. I too think it can be useful to allow stochastic potential outcomes. Indeed, I think the idea of the “stable unit treatment value assumption” (SUTVA) can be misleading, in that SUTVA requires that outcomes be modeled deterministically. In my above note, I said that any stochastic potential outcome can be expressed in reduced form as a deterministic potential outcome–but for modeling purposes, the stochastic model often makes more sense. So I guess there was something unclear in my paragraphs just above, in that I wasn’t doing a good job in conveying this perspective.

We interrupt our usual programming to inform you that I was never hired by Facebook to do a hit job on Jonathan Haidt.

Posted on by
32

So. I got some emails today from various people, pointing me to this item from a court filing:

This seems to be the story:

Meta internally set a goal to “get the academic community to disavow” and “turn against” professor Jonathan Haidt, who has linked social media use to mental health decline, as it in-house rebuffed internal efforts to study social media’s impact on teen health, according to recently unredacted documents filed publicly Friday. . . .

Notes taken during a March 8, 2023, meeting show Meta Vice President Curtiss Cobb’s suggested goal for the “Haidt assignment,” was, “how do we get the academic community to disavow Haidt?” Under the topic, a sole bullet point says “can’t co-opt them, so how do we enrage them?” . . .

The meeting notes also reveal that Meta employees discussed how the company could work with “the academics” to “get Gellman to write a take-down piece” on Haidt, referring to Columbia University statistician Andrew Gelman . . .

And this:

The employee wrote that Curtiss “thinks we need to focus on the media” to convince reporters that Haidt is “not credible.” Meta’s staffers also apparently discussed contacting a media figure referred to as “Gellman” to “write a take-down piece” about Haidt. It was not immediately clear who Gellman was.

It’s not immediately clear to me who Gellman is either!

But I will tell you this. I’ve never been contacted by anyone at Facebook or Meta regarding Jonathan Haidt or his work.

One of the above-linked articles mentions a court date from last year, in which “Meta abruptly settled the dispute, cutting a $17.1 billion deal with the states.”

$17.1 billion! OK, I wouldn’t expect all of that money, just for writing one little hit piece, but what if Meta had offered me $1.71 billion, or $171 million, or even just a trifling $17.1 million. Would I have done the job for them for that amount of money? I’d like to say, No, I’m too ethical for that, but . . . this a counterfactual about which we have no direct data, as they never approached me! My guess is that, had they dangled those dollars in front of me, I’d just have mocked them, kind of like how I mocked those Dialog 2022 scammers. But we’ll never know!

In all seriousness, this revelation about Facebook’s public relations team is absolutely horrible—we’re talking about some of the richest and most powerful people in the world conspiring to interfere with the open processes of science, journalism, and the law—but it’s hardly surprising, given what we’ve learned from insider accounts such as Careless People, the memoir by Sarah Wynn-Williams, or, for that matter, the despicable behavior of the bigshot legal teams in the Theranos and Cambridge University scandals.

Oddly enough, I have at times received money from Facebook; indeed, they’re listed on our sponsors page, along with other crowd favorites such as Disney, Amazon, and the U.S. Department of Defense. Back in 2016, Facebook gave me a no-strings-attached donation of $25,000 through Columbia University to spend on my research. This came from their data science team, which to my knowledge had nothing to do with the creepy behavior described above. I also spoke at their offices once (a technical talk, nothing about take-down pieces in the media), and they use lots of multilevel modeling, Bayesian inference, and Stan. Going through my old emails, it seems that we received some Facebook funding in 2020 for a computer science Ph.D. student who was working with me on a project involving probabilistic programming. We also applied for a couple of academic Facebook grants that we didn’t get. Also I gave Goel, Mason, and Watts some advice twenty years ago regarding their project on social networks that used a Facebook survey.

But nothing from them regarding social media’s impact on teen health. I guess that “get Gellman to write a take-down piece” was one of those bright ideas that never made it out of the conference room. Or else they approached a different person, maybe someone actually named Gellman.

A big problem with conventional experimentation or A/B testing is that the treatment effect can vary over time and it can depend on the entire history of the treatment. Usual comparisons miss this point.

Posted on by
12

Andrea Bronzini points to this article and writes:

I’m working on experimentation tooling and wrote a brief note about a failure mode I keep seeing: “winning” A/B tests often lose uplift after rollout.

My claim is that the typical setup mixes novelty response with steady-state effect because assignment happens after first exposure, and the outcome is time-dependent.

My reply: I think he is identifying a real problem. The problem he identifies is a special case of the issue that the treatment effect can vary over time and it can depend on the entire history of the treatment, and I guess that ultimately the best solution is to model these sources of variation.

I also like this bit from his webpage:

He saw how the industry’s obsession with certainty slowed businesses down… so he decided to fix it. . . .

Confidence ≠ Certainty.
Data is never complete.
And that’s why we need to embrace taking decisions with risk.
Progress belongs to those who act while others wait.

Also relevant is my talk (not yet a paper), It’s About Time.

“Death and Taxes: Taking Causal Inference into the Real World”

Posted on by
1

Nandita Mitra is giving a webinar this Tuesday at noon with the above title and the following abstract:

Statisticians develop methods with the hope that they will ultimately inform decisions and improve the world around us. But taking causal inference into the real world requires more than choosing the right estimator: it requires identifying the question that matters, engaging deeply with its context, and communicating statistical reasoning to very different audiences.

In this talk, I will draw on two experiences in which causal inference took me well beyond traditional statistical applications. I will first discuss my experience serving as an expert witness in a death penalty case involving racial disparities in jury selection, where ideas including confounding, mediation, and individual- versus population-level effects had to be translated into compelling and understandable arguments in a court of law. I will then describe my involvement in evaluating Philadelphia’s excise tax on sweetened beverages, where cross-border purchasing led us to confront interference and develop methods to better inform policy decisions.

Through these examples, I will reflect on how statisticians can seek opportunities to bring their expertise to consequential real-world problems, and on the importance of collaboration, communication, and stepping outside our usual academic boundaries. Most importantly, you don’t have to choose between being a methodological statistician and having an impact outside statistics.

This looks interesting. Click on the link above to register for the talk and get the zoom link.

JAMA rules of engagement on causal and predictive language

Posted on by
29

Erik, Witold, and I recently published a paper in the Journal of the American Medical Association. This was cool. A day after the paper came out, I ran into someone I know on the street and he said he liked our article. That’s kind of amazing, as usually it seems that nobody reads anything. We’ll have to publish in JAMA more often.

As part of the publication process, we were sent some general reporting requirements per JAMA standards. Here’s the part that will interest you:

USE OF CAUSAL LANGUAGE
Use of causal language (including use of terms such as “effect,” “efficacy,” “cause,” and “x increased y”) should be used only for randomized trials. For all other study designs (including meta-analyses of randomized trials), methods and results should be described in terms of association or correlation (eg, “x associated with an increase in y”) and should avoid cause-and-effect wording.

Avoid use of the words “predict” and “predictive of” for reports of observational studies assessing independent and dependent variables, unless your study design is a diagnostic/prognostic study and has used an appropriate approach to quantifying predictive performance such as (but not limited to) split sampling, cross-validation, or bootstrapping.

On the first item . . . Wow. I wonder if they’d let you write, “Cigarette smoking causes cancer in humans”?

I guess the second item is ok if it’s interpreted broadly enough. For example, Bayesian methods use an appropriate approach to quantifying predictive performance, so I guess that would be included in the “not limited to” category.

Getting better causal inferences by using more information (my talk at Stanford this coming Wed)

Posted on by
4

It’s the political economy seminar at Stanford business school, Wed 14 Oct at noon in room M109:

Lagged regression, difference in differences, two way fixed effects, regression discontinuity, placebo controls: these are different methods used when estimating causal effects from panel data. We’re working on a general approach using multilevel regression and poststratification (MRP) that aims to get the best from all these approaches. We demonstrate with a range of examples, including observational studies in economics, political forecasting, and controlled experiments in biology. This is joint work with Jori Korpershoek and others.

Come and ask me some tough questions!

The talk will be pretty open-ended. We have lots of ideas and some successes (for example, the chickens paper) but it’s mostly open questions and promising research directions. Should be fun.

I won’t be talking about this paper or this paper at all—that’ll have to wait until I’m invited to speak in the English department.

Hey—we’re hiring two postdocs!

Posted on by
3

I’ll post more formal ads soon, but, very briefly:

One project is on generalization for causal inference with a focus on education research, following up on MRPW: Regression, poststratification, and small-area estimation with sampling weights.

The other project is on Bayesian inference in scientific workflow with a focus on contamination models, following up on this work on design and analysis of bioassays.

Both projects are application-focused—that’s how we do statistics—but the importance of this work is not at all limited to these particular applications. This research will have relevance in policy analysis, medicine, and scientific measurement and inference more generally/

Further details soon.

When are Causal Inference Methods Needed to Answer Causal Questions?

Posted on by
2

Earlier this year we discussed Donna Spiegelman’s talk, “Rethinking SUTVA and Causal Identification: An Epidemiologist’s Perspective,” which she summarized as follows:

She’s giving a followup talk online on Mon 12 Oct, 3:30pm. Here’s the abstract:

When do specialized causal inference methods add value beyond standard approaches? Drawing from her unique perspective as both an epidemiologist and a biostatistician, Donna Spiegelman will consider the circumstances under which commonly invoked causal assumptions are necessary for causal inferences to be validly made from data, concluding that often not. She will show that valid learning can occur 1) under conditions much less restrictive than required by current widely used methods, 2) when real world implementation of interventions vary, and 3) when interventions spill over to others not directly exposed, thereby obviating components of the SUTVA assumption. She will provide evidence that measurement error is the major source of bias in observational research, not confounding, whose bias is rather tightly bounded. Finally, she will discuss the eternal challenge in science: after exhaustive efforts to collect data to predict important outcomes, a substantial proportion of the variation in occurrences of these outcomes appear to be entirely random.

The talk will be followed by a discussion. Click on the link to register (or show up in person if you’re at Yale).

Here’s a causal inference problem: where did the estimate of $64 million come from in “Blake Lively Says She Suffered $64 Million Financial Fallout From Justin Baldoni Conflict . . .”?

Posted on by
6

OK, here’s a causal inference problem for ya:

Blake Lively Says She Suffered $64 Million Financial Fallout From Justin Baldoni Conflict . . .

In Lively’s memorandum in opposition to Wayfarer Studios’ motion in limine filed April 17, her expert Dr. Ashlee Humphreys–a professor of marketing communications at Northwestern University–alleged Baldoni’s use of “the retaliatory phrases ‘tone deaf,’ ‘bully,’ and ‘mean girl'” during the alleged smear campaign against her had amounted to “176,738,781 impressions, in an amount of approximately $36.5 to $40.5 million.”

The filing also alleged that “retaliatory statements” made by the It Ends With Us director’s attorney Bryan Freedman during the legal battle had amounted to “116,959,530 impressions, in an amount calculated at $24,375,267.”

They continue:

Lively’s expert Humphreys had provided the calculation for reputational harm damages by quantifying the reach of the alleged phrases and statements, identifying “the number of impressions providing the basis for each” and calculating “particular dollar damages.”

I’m curious how the expert computed these dollar effects. I did some googling but couldn’t find the report with the numbers, so all I can do is speculate. Let’s do some divisions:

> c(36.5, 40.5)/176.738781
[1] 0.2065195 0.2291517
> 24.375267/116.959530
[1] 0.2084077

So, something like 20 cents per impression? I wonder where that came from. Maybe they did some sort of regression analysis estimating the effect of negative impressions on celebrities’ Q scores, and then attributed that to Lively not getting some roles that could have been worth $40 million and $25 million? The only thing is, the fact that both ratios are close to the same number suggests that they’re doing a straight-up calculation assigning a dollar cost per impression, but that can’t be right, as ultimately it has to result in foregone income, no? Also I don’t get why one of the estimates is a single number calculated to the nearest dollar value and the other is a range of round numbers. Also, once you have a range, I’m surprised it’s so narrow. Where does “approximately $36.5 to $40.5 million” come from? Why not “$25 to $50 million”?

If anyone has a copy of this expert report and would like to leak it to me . . . you have my email!

No joke–I’d like to see the analysis they did to justify their numbers. I’m not saying their estimated damages are wrong; Blake Lively is a big movie star, and her lifetime earnings could be huge. I’m just interested in the methodology here.

P.S. We’ve been assured by an expert from the National Academy of Sciences that scientific citations are worth $100,000 each. So now we can convert that to social media impressions (500,000 impressions = 1 citations) as well as more traditional cost benchmarks such as Jamaican beef patties, publications in Nature Communications, naming rights for a center for junk science, and, of course, attendance at a conference featuring Grover Norquist, Gray Davis, and a rabbi.

“Phonics is a fad, but it’s also good, so we should do it anyway, even though it will probably disappoint people”

Posted on by
32

Tyler Watts and Drew Bailey write:

Educational policy is often viewed as a field of fads, where one follows another for seemingly arbitrary reasons. We might like to believe that fad-based policy could be superseded by evidence-based policy, with rational actors choosing policies because of scientific evidence. But that’s been hard to achieve.

In the cynical version of the story, hucksters overpromise on their pet projects, we make the changes they advertise, and they get rich while the public is left disappointed. This probably happens more often than it should, but this narrative ignores the conditions that make such disappointment ubiquitous in educational policy.

Why? They explain:

Even if the hucksters all disappeared tomorrow, we’d still have this hype cycle, because overpromising seems inevitable when effects are small, and political disappointment will always follow. . . .

The educational policy research ecosystem usually selects for safe, often evidence-based, politically feasible, but ultimately oversold ideas, and we think that hurts our ability to make progress. In education, real effects are really small. . . .

Why are the effects so small? The outcomes we like to measure, such as reading comprehension, depend on many personal and environmental factors, making it hard for a discrete change in schooling to make much of a difference. . . .

This is a well-known problem in education research. For example, it came up in my 2018 talk at the Society for Research on Educational Effectiveness, and the point wasn’t new then either!

But, as Watts and Bailey explain,

The extreme education-nihilist position—that educational improvements can never affect students’ skills—is actually wrong. Endorsing it is likely to lead to poor education policy. The edu-nihilist position overlooks that small effects are not always zero effects. Small effects can actually be meaningful and, in some cases, worth it from a cost-benefit perspective for society.

One way to think about this is to consider classroom teaching.

I work my butt off every semester teaching my classes. Every class takes a lot of effort, both by me and the students. But any given class can’t have much of an effect on total learning. Indeed, the entire semester doesn’t have such a large average effect, when you consider that some students already could do most of it before the semester began, some still don’t get it when the semester is over, and others learn some things which they later forget. (On the plus side, students can learn all sorts of useful things that don’t happen to be on the exam.) But the effect of these classes is not zero. Students take them and they learn some things. So of course there’s no simple switch that will improve performance by 0.2 standard deviations. We’re already doing what we can. And the flip side of this is that a small improvement is not nothing. We teach each class knowing that at best it is giving a small average improvement, but it’s still worth doing.

Watts and Bailey continue:

When education policy researchers say something “works,” most people assume that effects are much larger than they are. . . . In most cases, researchers and policymakers fall prey to effect size magnification because they develop real sympathy for a pet program. When this happens, they put forward the most promising evidence, which is likely biased. The field then develops unrealistic expectations, and researchers, policymakers, and advocates downplay or withhold information about their uncertainty.

Hey, here’s an example from celebrated economist James Heckman, who doesn’t have much of an understanding of selection bias and goes around promoting wildly exaggerated estimates of early childhood programs.

Here are Watts and Bailey on what happens next:

Educational policies are uniquely poised to fall apart when overhyped, because public scrutiny of educational outcomes is so intense. Because most of the variance in student achievement comes from factors schools can’t control, overselling sets up an inevitable letdown. When disappointing results emerge and large gaps in achievement remain, we overreact to the apparent failure. Feeling the heat, the researcher, policymaker, or advocate might retreat and say that whatever was promised was never supposed to be a “panacea.” This always feels cheap because the language rarely shows up in the press releases, grant proposals, white papers, or speeches preceding the policy. Ultimately, we are left abandoning ship, even if the policy in question was actually helping.

The clearest example of this is probably the accountability reforms pushed by No Child Left Behind. Everybody hates No Child Left Behind because it was terrible and didn’t work! Right?

Actually, our best policy evaluations suggest it did work; it just had small effects, which is exactly what we should have expected. Instead of taking the small effects as a sign that something is working and figuring out ways to improve it, politicians and advocates decided it was a failure and moved on to other things.

And then they turn to the latest trend:

In a country that can’t agree on much, states from Alabama to California have adopted policies designed to promote the science of reading. . . . Count us as phonics believers. Kids need to know how to sound out words when they learn to read. We taught our own kids how to read this way, and we ourselves even learned to read this way in the pre-science-of-reading dark ages of the 1990s. But the unfortunate reality is that we have no reason to believe a sudden shift to a phonics-based curriculum will significantly affect reading outcomes in this country.

There are several reasons why.

First, meta-analyses of phonics interventions show fadeout. This means that kids who get phonics instruction for some period of time look a lot like kids who didn’t get phonics instruction in terms of reading just a year or two after the intervention ends.

Second, teaching phonics seems to help with decoding and word recognition. This is an important building block of literacy, but there isn’t much evidence that it helps with eventual reading comprehension—the ultimate goal. . . .

Finally, we are fooling ourselves if we actually believe that phonics hasn’t been used in classrooms up to this point. Do we really think that it never occurred to teachers in the “woke” states to help kids break words into syllables and sound them out? The counterfactual condition here has become a strawman. . . .

The best evidence on what happens when states adopt early literacy policies is probably this working paper, which finds effects of up to about .10 standard deviations . . . on high-stakes tests for 3rd graders (that’s good). They didn’t find much on low-stakes tests administered to 4th- or 8th-graders.

So, if we enter a halcyon period for phonics instruction, we might help a lot of kids learn basic decoding skills when they are first learning to read. But . . . effects will be smaller than we were led to believe. Importantly, they probably won’t move the needle much on outcomes like NAEP trends, which compare U.S. cohorts of 4th and 8th graders to each other. . . . The question is: how will we interpret this when the sobering news starts to come in?

Just in case we weren’t clear: doing more phonics in early-grade classrooms is, for the last time, probably a good thing.

They conclude:

Education policy is not the only field that faces conflicts between public and expert opinion. But the combination of big expectations, small effects, and constant attention to children’s educational outcomes means education policy is likely to keep experiencing these conflicts.

What can we do to mitigate this educational political economy problem? . . .

We should ask interested parties to make predictions on the record about how these children will differ (and by how much). And our society should reward researchers and policymakers whose mental models are closer to reality, rather than those who are best at selling potential solutions in the first place.

And my own thoughts

I agree with everything that Watts and Bailey wrote above, and also have a few things to add:

First, there’s my recent paper with Amy Krefman, Lauren Kennedy, and Jessica Hullman, Hypothesizing an effect size by considering individual variation, where we try to think systematically about how to get more realistic expectations of average treatment effects.

Second, a point from my above-linked SREE talk, that research should be less isolated from practice. From one direction, research studies should be more realistic, more connected to the real world of education—this is a point that Watts and Bailey make in their post. From the other direction, general practice should follow principles of research: students, parents, teachers, and administrators should be empowered to make choices and experiment, and, along with this, choices and outcomes should be measured so that effectiveness can be studied. We should all be trying to do better anyway, and we should design our data collection so we can all best learn from these experiences. As we discussed in a different context, local data collection, collaborative analysis, and local recommendations.

Third, ummm, this is a longer one so I’ll just quote myself:

You can conceptualize an education intervention as a vector, where the direction of the vector is the material being learned and the length of the vector is the amount that students are motivated to work. You want the material learned to be useful—you’d like the vector to have a positive “dot product” with the vector of skills, knowledge, and understanding that will be useful going forward—but, conditional on those two vectors being roughly aligned, the real gain is in the magnitude. And this magnitude will be an interaction between the teacher and student: there’s no button to push to create motivation, and if there were a button it would already have been pushed.

What I’m saying is that, when thinking about acupuncture, or physical therapy, or coaching, or teaching, we have to go beyond what I’ve called the penicillin model of science, the idea that innovations come from nowhere and that the job of statistics is to design and analyze experiments to reject the null hypothesis of no effect, and in which the treatment in such experiments is considered as a black box, with the goal being to estimate an average treatment effect.

I don’t think the penicillin model usually applies. Most of the time in health, education, and just about any field, improvements are incremental, and the goal is to improve the process while gaining understanding. Clinical trials and offline experiment both play a role, and you’re not going to learn much by studying a treatment as if it’s a black box.

This is not to say that there cannot be new developments in any of these fields, nor is it to deny that such developments can sometimes arise serendipitously. I just think that, in any case, you have to go beyond the average treatment effect and think about the mechanism of action.

OK, here’s a statistics problem for you, ripped directly from the headlines!

Posted on by
21

From the NYT:

Mayor Zohran Mamdani is expected to propose linking Grand Army Plaza with Prospect Park by closing a dangerous stretch of road between them. . . .

The plan would effectively reconnect the oval-shaped plaza’s most prominent feature, the Soldiers’ and Sailors’ Memorial Arch, with Prospect Park, Brooklyn’s emerald jewel, restoring the 80-foot-tall arch as a grand entrance to the park, as its designers intended.

Cars would be banned from the parts of the southern end of the plaza that border the park, from Union Street to Eastern Parkway, which would do away with a forbidding four-lane crossing, according to the city Department of Transportation. . . .

After public feedback, much of which favored more dramatic changes to the area’s roadways, the Mamdani administration selected a plan that would remove cars from the lower tip of the plaza and divert some traffic to other streets.

They quote people on both sides, who raise a number of points:

Lynda Balsama, an organizer with . . . United Neighbors of Prospect and Crown Heights, said the redesign would divert more traffic to nearby residential streets and add travel time for fire trucks and ambulances. . . .

Between 2021 and 2025, there were 219 traffic injuries along the plaza’s central roadways and outer ring, according to the Transportation Department. . . .

In spite of the constant traffic, the plaza remains a popular destination for those who want to admire the work of Frederick Law Olmsted and Calvert Vaux, the designers of the plaza and Prospect Park (as well as Central Park in Manhattan). . . .

“Every time N.Y.C. D.O.T. has provided more space to pedestrians at the park, it’s been an instant success, and it becomes impossible to think of how the space could have functioned before,” Mike Flynn, the transportation commissioner, said in a statement about the plan. . . .

The Transportation Department also expects the plaza redesign to help speed up some of the slowest buses in the borough, including the B41, which serves over 27,000 daily riders, because the reconfiguration should reduce congestion in and around the loop.

Here’s the official announcement from the city government.

Here are the issues raised above, stated more abstractly:
1. Current public opinion,
2. Worse traffic on other streets,
3. More fire damage (because the fire trucks are delayed) and more deaths or serious health problems (because the ambulances are delayed),
4. Fewer traffic injuries at the intersection in question,
5. Happier tourists and locals who want to enjoy the park,
6. Track record that past pedestrian expansions have been popular,
7. Faster buses.
I’m sure you could think of more; this is what I got from the news article. Of these seven issues, five are pro-redesign and two (#2 and #3) are anti. That doesn’t mean the plan is a good idea, as it reflects the balance among who was quoted. That said, I did some googling and it’s hard to find much opposition to the plan, so maybe the Times had to work to get even that one quote.

In any case, the research challenge is to put numbers on all these things. This still doesn’t answer the question of what to do, as it involves balancing questions of public opinion, economics, convenience, and life and death. But it would be a start. It’s hard for me to imagine that item #3 counts for much at all, but it should be possible to give some estimate of the components that go into it. Indeed, it could be that the fire trucks and ambulances would actually come faster.

Anyway, it seems like a good example, to work through the numbers on the seven items above.

Metascience corner: What can we learn from 12 million empirical research results?

Posted on by
6

He approves

This is Witold and today we are building a bear.

In our recent paper with Erik van Zwet and Andrew, A statistical case for qualified scientific optimism, we fit metascientific models that use hundreds of thousands of z-values from empirical research. We will be blogging about this paper in the coming months, but the data that we needed for this paper seemed so interesting, that they spun off into a whole separate resource, Benchmarks of Empirical Accuracy in Research. I think readers interested in metascience will find it useful.

BEAR is an open-source compilation of 26 (and counting!) documented metascience datasets. Sometimes it just re-uses data from publications, like Kevin Lang’s work on false positives in economics—and dozen more. (I can’t stress this enough, huge thanks to the legion of researchers who created/maintain these datasets.) For some others I do much more myself, like processing outcomes from tens of thousands trials from clinicaltrials.gov.

And you can access all of that, 12.5 mln results in total, with a few clicks. Just go to https://witold.xyz/BEAR/. BEAR has bespoke datasets from medicine, neuroscience, psychology, ecology, education, economics, political science and more—all standardised and ready to use. Some sets have only z-values, but some have much richer structure: study types, subcategories, effect sizes, type of measures, meta-analytic groupings, etc. Of course quality varies, but I hope the documentation/standardisation will help people pick the right dataset for their research question.

Why does this exist? As I say on the website:

Quantitative metascience drives important debates about research standards. Most of its crucial contributions have been based on analysing individual datasets, often painstakingly constructed by researchers. Thanks to their work we now have better understanding of replicability, publication bias, p-hacking, reporting practices, pre-registration, significance rates, and more. But many canonical findings of metascience are based on individual datasets, each constructed in a specific way. How generalisable are these findings?

In other words, metascience itself can suffer from one of the crucial metascientific concerns: external validity. As someone who works on evidence synthesis I am very concerned with this. To give one example, Erik blogged about the virality of the scary publication bias histogram in biomedical journals. But different metascientific sets give completely different pictures of selection:

I am NOT particularly interested in selection in this post. We could talk about any other metascientific property, e.g. p-hacking, low power. I am also not trying to explain what drives these differences or say that these sets are comparable. (We get into that a little bit in the paper and if people are interested in that particular question, I can write a follow-up post.) The point is that we can learn much, much more when we go beyond one data point. OK, that’s pretty trivial thing to say, but Erik’s example clearly shows that it’s one that hasn’t fully sunk in. So I just hope that BEAR will make it easy to compare metascientific corpora and give people a better picture of the limits of generalisability of various claims. Or generally make metascientific research easier.

For completeness, the GitHub repo also includes the model that Erik, Andrew and I use in the paper I linked earlier. You can use it yourself to e.g. estimate (idealised) expected replication rates, but the model is optional and separate from BEAR data. Big kudos to Erik who got this process started and hunted down many of these sources for the paper. But most importantly the credit goes to teams of researchers who generated and maintain the linked sets.

I also have no doubt that in putting this together I made many mistakes. Even with automated checks, it’s not easy to work with so many data sources in parallel, so I’d be grateful for any corrections!

Update on a regression discontinuity dispute: some asynchronous collaboration

Posted on by
16

Anjali Thomas writes:

I am writing to share a paper which is a re-examination of my 2018 AJPS article entitled “Targeting Ordinary Voters or Political Elites”  which was previously discussed on this blog here.

The paper, written in the spirit of Gelman (2022), conducts a thorough re-analysis of my earlier work in light of recent developments in regression discontinuity designs (RDD). It also directly addresses specific critiques of the article raised both in subsequent academic literature and in previous comments on this blog.

A full response to each critique raised on this blog appears in Section A.3 on page 48 of the paper. Among other things, the blog critiqued the use of the global fourth order polynomial, and commenters highlighted that it appeared to be picking up noise in the data rather than a true relationship. While the original article did present results in the SI showing robustness to a local-linear specification with alternative bandwidths, the currentpaper significantly extends these checks and presents new results showing:

  • Polynomial & Bandwidth Robustness: Shows stability across lower-order polynomials, alternative bandwidths, kernel choices, and a donut-hole approach.

  • Noise Reduction via Aggregation: Aggregates data to the level of the running variable to reduce noise, presenting new scatter plots where the visual discontinuity persists at this level of aggregation.

  • Spatial Structure: Demonstrates that reported balance/placebo issues in recent re-analyses stem from ignoring within-constituency clustering and predictive controls.

A link to the paper is here (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7272598) and the abstract is below:

This note reexamines Thomas (2018), which advances and tests the ‘elite cooperation logic’ whereby national politicians target resources along partisan lines to win over the cooperation of co-partisan state legislators in implementing development projects. Consistent with this logic, a close election regression discontinuity design (RDD) uses project-level data to show that national legislators in North India allocate systematically higher public works expenditures to constituencies of co-partisan state legislators in the period after a state election. Responding to critiques in subsequent literature of the RDD approach used, this note shows that both the evidence of covariate imbalance reported in Bicalho et al. (2026) and the high proportion of significant placebo estimates reported in Albada (2025) are artefacts of ignoring within-constituency clustering and omitting predictive controls. Meanwhile, this note presents re-analyses confirming that the core findings in Thomas (2018) are robust to covariate inclusion using either constituency-level clustering or constituency-level aggregation. Acknowledging the problems related to global fourth order polynomials (Gelman and Imbens, 2019; Albada, 2025), the results in Thomas (2018) are also shown to be robust to lower-order polynomials, alternative bandwidths, alternative kernel choices, the donut hole approach, and an inference procedure that adjusts for worst-case bias (Stommes et al., 2023). The note re-establishes the credibility of the substantive findings in Thomas (2018) and highlights the importance of accounting for spatial clustering in both estimation and diagnostic checks in RDDs.

Anjali took my Bayesian statistics class back in 2006! It’s great to see what former students are doing, and I love seeing this sort of asynchronous collaboration.

Regarding the regression discontinuity issues, I do not think it makes sense to fit unrelated curves on the two sides of the cutoff. So I prefer the versions that fit a single curve plus discontinuity, not two curves.

The other thing I always recommend (see this recent paper with Imbens is that regressions include not just the forcing variable but also other pre-treatment predictors. This should help with both bias and efficiency.

All the discussion of bandwidth, functional forms, etc., can be wasted if the fitted model makes no sense or if it’s a distraction from including additional pre-treatment predictors.

In any case, it’s great to see this sort of open exploration.

I recommend this book on Applied Regression and Causal Inference

Posted on by
5

This post offers two recommendations and a story.

First, the recommendations.

• Here’s the book referred to in the title of the post. I highly recommend it! My only regret is that we didn’t call it Applied Regression and Causal Inference, cos that’s what it’s about.

• I also recommend our book, Active Statistics, which has literally hundreds of stories, class-participation activities, computer demonstrations, and discussion problems. A great teaching resource and also just a good read. Really!

Now, the story.

This came in the email one day:

Dear Professor Andrew Gelman,

I hope this message finds you well. I’m part of the Cambridge University Press sales support team and I work with your local Cambridge Sales Representative ** **@cambridge.org

We wanted to follow up regarding your course(s) for Fall 2025 and ensure you’ve had a chance to review Regression and Other Stories, 1 ed by Andrew Gelman/Jennifer Hill/Aki Vehtari as a potential option for your course materials.

If you’ve decided to adopt this title, I’d be happy to provide access to any available instructor resources. If you’ve chosen a different title or are still considering your options, I’d appreciate a quick update so I can ensure our records are accurate.

Also, please don’t forget to check with your bookstore about any Equitable or Inclusive Access programs that may be available. These programs can provide your students with more affordable digital access to their course materials.

I am sincerely grateful for your attention to this matter; I look forward to the opportunity to hear from you at your earliest convenience.

Cordially,

**
Sales Support Assistant
Higher Education Sales, Americas
**

On one hand, hey, I’m thrilled that they’re promoting our book. On the other hand, it seems like they’re kinda flying blind if they’re sending this promotional message to the author of the book.

“Placebo tests deserve a model, not just a glance.”

Posted on by
1

Miha Gazvoda shares this post with the above title and the subtitle, “Using Bayesian multilevel models to correct bias and calibrate uncertainty.”

He’s using the chickens model from our Slamming the Sham paper in the more general setting of placebo control tests.

In econometrics, a “placebo control test” does not need to literally involve a placebo treatment; it more generally is used to describe a procedure in which the same statistical analysis that was used to estimate a causal effect is applied to a different dataset, or a different part of the existing dataset, in which the treatment did not occur.

For example, if you want to measure the effect of an intervention that occurred in 2021, you could repeat the analysis but using data from a different year. Or if you want to measure the effect on a particular outcome, you could repeat the analysis but looking at a different outcome that should be unaffected by the treatment.

The idea is that, if your estimation method has an artifact or systematic bias, this should show up in the placebo analysis as well, indicating a problem. Conversely, if the placebo analysis does not show an effect, this is taken as evidence that there is no artifact.

In his post, Gazvoda argues that, rather than using the result of the placebo check to make a go/no-go decision, it should be possible to partially adjust the treatment effect to account for the information in that supplementary analysis.

This makes sense to me, and of course I’m happy that he’s using our chickens model.

There’s a tricky thing going on here with the placebo check, which, interestingly, arose in the chicken example too, and that is that we usually don’t have any good theory for where the effect is coming from in the placebo control. After all, if our causal identification is working as designed, we shouldn’t even need the placebo comparison, as we’re already getting an unbiased estimate of the treatment effect. The placebo control is typically there to address unspecified concerns of bias. And, indeed, in practice, researchers don’t always do placebo controls. And when, as hoped, the placebo control shows no statistically significant effect, the inclination is to take that as a reassurance and move on, in the same way that is done with other robustness checks.

The lesson Gasvoda takes from the chickens example is that if replications are available, you can assess the evidence for the placebo adjustments being relevant: you can fit a multilevel model to estimate how much adjustment needs to be done.

From a sociology-of-science point of view, it’s interesting to me that the conventions in biomedical statistics and econometrics go in opposite directions:

– In biomedical statistics, the default recommended behavior is to compute the difference in differences, taking the estimated effect from the main experiment and subtracting the estimate from the placebo experiment. As we explain in the chickens paper, this correction has the disadvantage of doubling the variance of the estimate, a true statistical crime if, as is often the case, the effect of the placebo treatment is indistinguishable from zero.

– In econometrics, the default procedure, if you’re calling it “difference-in-differences estimation,” is the same as above. But if you’re calling it “placebo control,” and the placebo estimate is not statistically significant from zero, the default is to ignore the placebo results entirely, not to adjust for them.

In general we recommend a partial adjustment, with the amount of adjustment depending on the problem at hand. If there is internal replication, as in the chickens example, the appropriate adjustment factor can be estimated from the data. If it’s a one-shot experiment, you’ll need to use prior information. I don’t have any good examples demonstrating how to do that; it’s something we should do.

What gets you is not what you don’t know but what you don’t know you don’t know.

Posted on by
2

I was thinking about the above saying in the context of bad regression discontinuity analyses.

Statistical methods can be characterized in terms of how they can go wrong. Some common modes of “how can things go wrong” include:
– biased measurements,
– selection bias into the observed dataset,
– differences between treatment and control groups,
– nonstationarity or, in general, differences between sample and population of interest,
– biased or noisy statistical estimates,
– missing data,
– problems with functional forms (for example, using a linear model for a nonlinear relation, or not including important interactions),
– problems with error terms (dependence, choice of distribution, etc.),
– unmodeled spillovers, hierarchical structure, or other violations of causal model assumptions,
– researcher degrees of freedom and forking paths,
and lots more!

Regression discontinuity is used for observational studies, and the #1 thing that can go wrong in an observational study is #3 on the above list: differences between treatment and control groups.

The problem is that when researchers perform regression discontinuity analysis, they focus on the problem with the functional form of the expected outcome given the one predictor, often ignoring other potential predictors even if they are screaming to be included (as with age in this example). There’s all this obsessing over the functional form (and I guess Guido and I contributed to this) that’s kind of missing the point (as we discuss briefly here).

So, the problem is not that these users of regression discontinuity don’t know what to do with other predictors; it’s that they don’t know that they don’t know this.

As Tolstoy might have said had he been a data analyst, Models that fit are all alike; every poorly-fitting model is poorly fitting in its own way.

P.S. Here’s some background from the Quote Investigator on various version of the saying that I used as the title of this post.

Why quantitative understanding of effect sizes matters, even if all you care about is the presence of the effect

Posted on by
16

In reaction to my article with Andy King proposing post-publication review, Dan “Fast and Frugal” Goldstein writes:

Your process limits information search, computation, and time so it seems fast and frugal to me. Happy you still associate me with that term. It was something I coined as a grad student.

In your proposal, only hit papers get audited. It reminds me a bit of Mel Brooks’ The Producers in which the protagonists use the logic “who would audit a flop?” and stay under the radar by intentionally producing a bad show. Fraudsters have likely attempted to make their work seem worthy of publication while ensuring it doesn’t attract too much attention. Just like in The Producers, though, this sometimes backfires.

I don’t know about that! My impression with fraudsters is that they think that fraud is normal science, perhaps out of some mixture of bad education in research methods, a view that “everybody does it,” and a general lack of understanding of how non-cheaters (like you and me!) think. Think of people like Wansink who gave general advice to to the world on how to p-hack, or Gino and Ariely, who published papers on dishonesty, or Mary Rosh, who surely believes that whatever shady statistical manipulations she does are nothing compared to the dastardly deeds done by the Democrats.

I’m sure there’s tons of below-the-radar cheating and bad science that we don’t hear about, but a fair number of prominent science fraudsters seem to enjoy the limelight. One reason for this seemingly self-sabotaging behavior, I think, is that cheating enabled these people to attain great professional success for years. They had no reason to think the juice would stop flowing.

To return to my proposal with Andy King: I think it’s ok that only the hit papers get audited. Bad papers that get no intention aren’t doing much damage, right?

Goldstein adds:

By the way, I was just having a conversation about your sensing that something was amiss with the LaCour study. For years now I have used this quote of yours in a talk I give about putting numbers into perspective. I argue that it’s really important that people learn how to put numbers into perspective because if they don’t, they won’t notice that something is unusual and worthy of a deeper audit. You somehow sensed something was up with the Lacour result. You didn’t think it was fraud yet but you knew it was strange because you know how to put such differences into perspective:

A difference of 0.8 on a five-point scale . . . wow! You rarely see this sort of thing. Just do the math. On a 1-5 scale, the maximum theoretically possible change would be 4. But, considering that lots of people are already at “4” or “5” on the scale, it’s hard to imagine an average change of more than 2. And that would be massive. So we’re talking about a causal effect that’s a full 40% of what is pretty much the maximum change imaginable. Wow, indeed. And, judging by the small standard errors (again, see the graphs above), these effects are real, not obtained by capitalizing on chance or the statistical significance filter or anything like that.

My colleagues and I recently wrote a paper on this general topic of average effect sizes. It’s our contention that people generally are way too optimistic about possible effect sizes, in large part because they don’t think about variation. If you ask someone to hypothesize an effect size, you’ll typically get a guess of the largest effect that might occur.

But what if you don’t really care about effect size–you just want to know about the effect?

For example, maybe you don’t believe that women during certain times of the month are three times more likely to wear red or pink shirts, but you are interested in some sort of evolutionary psychology theory of sexual display. In that case, why should the effect size matter? Why care that a study reported an estimate that was ridiculously implausible?

I have two to this questions, and thus two reasons why effect size is important even for problems where you don’t directly care about effect sizes:

1. Effect sizes vary. An treatment that has an effect (that is, a true effect, not just an estimated effect) of 0.1 for one group of people in one setting could have an effect of -0.2 in some other scenario. A treatment effect in an experiment is the sum of all sorts of things, positive and negative, and there’s no logical reason to think the sign of the effect will be preserved. Effect size matters. The issue is not just that a smaller and more realistic effect size is less important; it’s also that smaller effects can be more easily produced by other factors, and this reduces the generality of any claims, even if the experiment at hand was done well.

2. Experiments produce standard errors as well as estimates. If the standard error from a study is large compared to any realistic effect size, then the study contains very little information. Effect size is important in understanding the informativeness of an experiment, and to do this right you need to have some sense of what the true effect size could be. You can’t just use a point estimate from the study itself, as this estimate will inherently be too noisy to use to judge the information in the study. As I wrote in this note for the Annals of Surgery, Post-hoc power using observed estimate of effect size is too noisy to be useful.

Survey Statistics: equivalent models, equivalent weights (locally)

Posted on by
4

Last month we saw that the Times/Siena Poll is now using energy balancing weights (Huling & Mak, 2024). In a toy example, we saw under which outcome models these weighting methods might do well. I was inspired by Little 2004, who saw under which outcome model the inverse-probability-weighted estimator (a.k.a. Horvitz-Thompson) does well. This explains the HT estimator’s poor performance in Basu’s (1971) elephants example (which I’ve used in my post Basu’s Bears).

From Little 2004:

Andrew’s 2007 “Struggles” paper and the 2026 MrPlew paper go the other way: start with an outcome model and back out the weights. These folks all worked pretty hard. I wondered if I could just use Thomas Lumley’s survey package to get the weights. See my posts “struggles with equivalent weights” and my continued struggles. But in the simulation I used a linear outcome model. It would have been more interesting with a logistic outcome model !

The linear case was handled by Andrew’s 2007 paper. The 2026 MrPlew paper extends this by noticing that the linear equivalent weights are derivatives, and using this to define locally equivalent weights, green highlighting by me:

Then the 2026 MrPlew folks use these equivalent weights to compare the target population to the weighted sample, as an MrP diagnostic. (Unlike the Huling & Mak paper on energy balancing weights, the MrPlew folks look at one covariate function at a time, not the entire covariate distribution at once.) One of their examples is from Lax & Phillips 2009 Gay Rights in the States, which I blogged about for pride. In this example, the implied covariate (im)balance for MrP looks not great ! Here’s their Figure 5b:

As Andrew blogged: “It makes sense that implied covariate balance can sometimes be worse for MRP than for raking. MRP is a smoothed version of raking, and unsmoothed raking can overfit.” Yes, but here it’s worse than the raw uncorrected balance ! The MrPlew authors caution here that the locally linear approximation may be extrapolating poorly, and that we shouldn’t over-rely on this model check. They also say that “logistic regression generally balances the variance-weighted covariates, but not the covariates themselves”. I am glad they included this illustrative example and I want to think more about it.

Thoughts ?