JAMA rules of engagement on causal and predictive language

Posted on by
10

Erik, Witold, and I recently published a paper in the Journal of the American Medical Association. This was cool. A day after the paper came out, I ran into someone I know on the street and he said he liked our article. That’s kind of amazing, as usually it seems that nobody reads anything. We’ll have to publish in JAMA more often.

As part of the publication process, we were sent some general reporting requirements per JAMA standards. Here’s the part that will interest you:

USE OF CAUSAL LANGUAGE
Use of causal language (including use of terms such as “effect,” “efficacy,” “cause,” and “x increased y”) should be used only for randomized trials. For all other study designs (including meta-analyses of randomized trials), methods and results should be described in terms of association or correlation (eg, “x associated with an increase in y”) and should avoid cause-and-effect wording.

Avoid use of the words “predict” and “predictive of” for reports of observational studies assessing independent and dependent variables, unless your study design is a diagnostic/prognostic study and has used an appropriate approach to quantifying predictive performance such as (but not limited to) split sampling, cross-validation, or bootstrapping.

On the first item . . . Wow. I wonder if they’d let you write, “Cigarette smoking causes cancer in humans”?

I guess the second item is ok if it’s interpreted broadly enough. For example, Bayesian methods use an appropriate approach to quantifying predictive performance, so I guess that would be included in the “not limited to” category.

Columbia University says, “We believe in difficult conversations at Columbia—in a broad embrace of freedom of expression. It’s in our DNA.”

Posted on by
23

Claire Shipman, the president of Columbia University, writes:

We believe in difficult conversations at Columbia—in a broad embrace of freedom of expression. It’s in our DNA. I intend to uphold that tradition for several reasons.

First, as a nation, we should be able to tolerate ideas, facts, and opinions that we don’t like. Tolerance is essential to building resilient individuals and a resilient democracy.

Second, it’s our mission. We have a responsibility to guide students toward intellectual rigor, to enable them to grapple with, and examine carefully, ideas different from their own. More broadly, a willingness to test assumptions is essential to our academic excellence in every field.

Third, and perhaps most important, universities—Columbia in particular—are uniquely positioned to help change the national tone. We can guide and model the art of examining complex, potent issues in a deliberate, respectful fashion. . . .

Our instinct might be, in such an era, to shy away from challenging events and speakers. The better course is to model how to do it well . . . Our statutes lay out quite clearly and sensibly that expression, on our campuses, does have limits, in particular as regards harassment, discrimination, or the threat of violence. Peaceful protest, civil discourse, a respectful exchange of ideas—this is our inheritance at Columbia, and I want to ensure that it is also our legacy. . . .

So, as this week begins, and as we continue to debate the affairs of the world and this nation on our campuses, consider this: at times there will be speech that each of us finds disagreeable, even abhorrent—and we still need to protect it. Maybe even seek it out. Such a practice makes us stronger.

I agree. And, although our president did not say so in this message, this policy of protecting and welcoming dissent, even when it is controversial or disagreeable, runs counter to some actions of our university in recent years:

  • The delayed reaction to the sexually-assaulting doctor in the medical school, without any seeming consequences to the people at the university who enabled this behavior.
  • The fake U.S. News numbers, where, again, there seems to have been very little interest by the university administration in finding out who did it and why. As with the medical school scandal, the avoidance of any serious investigation does not seem consistent with the goal of promoting free speech.
  • Barnard College, a division of Columbia University, attempting to cancel the showing of a controversial film on campus. At that time, it seemed that everything was about avoiding offending anybody and about avoiding any controversy—really the opposite of the view that “we should be able to tolerate ideas, facts, and opinions that we don’t like.”
  • Barnard College “mandating that students remove any items affixed to room or suite doors” under the rationale that they “do what we can to support and foster the respect, empathy and kindness that must guide all of our behavior on campus.” Empathy and kindness are fine, but, again, it contradicts the idea that “there will be speech that each of us finds disagreeable, even abhorrent—and we still need to protect it.”

As I noted the other day, Barnard College has also announced a new policy promoting free expression, and I think it’s a good thing.

I completely agree with the president of Columbia that “we should be able to tolerate ideas, facts, and opinions that we don’t like,” and I completely agree with the president of Barnard that “Colleges and universities need to reconfirm our commitment to nonviolent forms of disagreement — even when we are confronted with voices that disparage or dismiss identities and worldviews. This is also a time to foster more disagreement, not less.” I was getting very frustrated with all the happy talk and the official attitude that we shouldn’t be disagreeing. The university—and, of course, the country—has people who disagree with me on important issues, and people I don’t respect at all, for moral, intellectual, or political reasons. And I guess lots of those people don’t respect me either! Which is too bad, but we still have to exist, and we should have the right to express our views, both publicly and privately. We shouldn’t be fired for our nonviolent acts (except to the extent that they interfere with our ability to do our jobs), nor should be discouraged by the university from expressing them.

And, just to be clear, no one at Columbia has ever suggested or even requested that I keep my mouth shut in class or outside of class, including when I’ve criticized the government or the Columbia administration or anything else. I have nothing to complain about on my end, but I do think it was a mistake for them to crack down on dorm room doors, to try to stop the showing of controversial movies, and to keep quiet the bad behavior regarding the fake statistics and the terrible actions at the medical school.

I’m happy about this new policy and I hope the administration follows up by changing the dorm room door policy, apologizing for trying to suppress the movie, and starting and publicizing independent investigations into the U.S. News numbers and the enabling of the sexual assault at the hospital.

Also let me emphasize that by bringing up these earlier cases, I’m not criticizing the Columbia president Shipman, as these things were done before she took office. I also understand the desire to move forward rather than looking back; still, I think that a great way to reinforce the policy of enabling difficult conversations is to revisit some recent examples where the university had unfortunately avoided doing so.

“It’s in our DNA.”

One thing that jumped out at me when reading the president’s email was the statement that freedom of expression, especially in difficult conversations, is “in our DNA.”

My first reaction was the annoyance of the persnickety social scientist: calling it “DNA” is just wrong, and it indeed muddies the waters, as we’re talking about something social, not genetic.

But that reaction of mine was silly. Obviously no one when using “DNA” in that sense is taking it literally; it was a metaphor. And maybe we can gain some insight by taking this metaphor seriously.

For something to be in an organization’s “DNA,” that would imply that it’s not just a transient trait; rather, it has to be in some equivalent of self-replicating code. The idea is that freedom of expression and willingness to confront uncomfortable ideas are not just attitudes that sometimes has existed at Columbia, but rather that they are embedded deeply enough that they will persist. Indeed even if free inquiry and expression at Columbia are suppressed (as with the sexual assault, the fake statistics, the dorm-room censorship, and the attempt to suppress the film screening), the self-replicating aspects of our university’s rules and culture will cause them to return.

And, from that perspective, I do see something in our president’s analogy to DNA. Columbia has for many years celebrated the legacy of the 1968 protests, while recognizing that the other side at the time had legitimate concerns too. That is, Columbia has recognized the value of protest without implying that any particular protest action has to be recognized as being a good idea. We have the University Senate, which dates from that period, as well as generations of new students and staff who have exercised the freedom of dissent—even dissent that some can find disagreeable—and recognize this as a Columbia tradition.

Indeed, this recent statement by the president can be viewed as a consequence of that DNA, an expectation that our university is a place with diverse enough attitudes that there will be people who disagree with and even despise each other but still have the right and, at times, the obligation, to air these uncomfortable disagreements publicly.

Because the alternative, of simulating a nonexistent harmony, would be intellectually dishonest and politically dangerous.

The problem is noise, not N: why increasing the sample size isn’t all that.

Posted on by
10

Someone was talking with me about the problem with lots of published social science research that’s based on small sample sizes. And I agree that this is a problem! But I think the real problem is noise, not N.

Noise (variation in effect size, variation in outcomes, and measurement error) is more of an issue than sample size because lots of bad studies are so noisy that, to get a large enough sample size to detect a reliable effect, you’d need N = a zillion, and at that point you’d be averaging over such a wide range of conditions that it’s not clear what you’re estimating anyway.

There’s a widespread misunderstanding by which people think that their problems can be solved with bigger samples. I think the erroneous reasoning goes like this: People are aiming for 80% power, but they really have only 50% power, or 30% power, or something like that. If you have 50% power and double your sample size, you’ll have 80% power . . . a win! The error in this reasoning is that often the power is much less than 50%, more like 5% + epsilon. The flip side of this is that if you really are running studies with 50% power or even 30% power, you’re likely to get the right direction on the effect size anyway, so often there’s no good reason to wait on statistical significance.

So, lots of moving parts here: There’s the underlying effect size, the variation in effect size, the variation in the measured outcomes, and researchers’ expectations when setting effect sizes and interpreting results.

Digital twins version 2.0: The skeptics speak up

Posted on by
14

This is Jessica. Several years since the earliest research papers on the topic came out, the idea of using LLMs to simulate human behavior like survey responses or experiment outcomes (“silicon sampling”) is hitting mainstream. People tend to bring this up in conversation with me, probably because I’ve written about validation and taught a course, so I get to hear a lot of other researchers’ views. I would put the majority of takes I encounter in either the more agnostic, “Somebody should probably explore how much we can do with them” and/or “I’m pretty doubtful it’s going to live up to the hype” camps.  

One way to use silicon samples is digital twinning, where simulations are conditioned on considerable information about specific individuals, and expected to predict the responses of those specific individuals. Early papers on digital twins sing their praises, showing for example that they can reproduce people’s responses more than 80% as well as people replicate their own responses (e.g. here or here). But we’re now seeing a more critical empirical push, which uses novel tasks and topics and more careful design of metrics and baselines for comparison to gauge how much conditioning on more individual-specific information improves prediction. 

In the last month, there have been a few noteworthy data points. Pew Research put out a report describing what they learned from comparing digital twins implemented by giving Claude Opus 4.6 detailed 2025 survey responses and demographic persona information for people who responded to several more recent survey waves. They compute the average absolute error in percentage points for each multiple choice survey question (i.e.,the average of the absolute differences in the proportion of LLM versus human respondents selecting each answer option), and find it averages 12 percentage points over the 300 survey questions they looked at, with 28% of the questions having error over 15 percentage points. They observed several kinds of biases that have come up in other research comparisons: less variation in LLM responses compared to human (with some answer options never being picked by the LLM), stereotyping (like predicting that 97% of Hispanic adults are at least somewhat likely to follow the World Cup; in reality its only 43%), overestimating human accuracy on factual questions, and mispredicting sentiment on current events like data centers and face coverings on ICE officers. They stress how hard it is to predict how far off the LLMs will be on a given survey question.  

This list of biases closely resembles those proposed by Tianyi Peng et al. in one of the most comprehensive evaluations of digital twins to date, recently published in Science Advances: Digital Twins as Funhouse Mirrors: Five Systematic Distortions. They collect a new set of 19 preregistered studies involving 164 outcomes, and find that overall, there is only weak improvement over base models and relatively low correlations with human behavior (on the order of 0.2). 

I like this paper a lot, in particular the comparisons they do to simpler demographic prompting and base models. The intuition that has been sold by researchers and companies in the space alike is that with rich enough information about individual people, we can predict their behavior with high accuracy. To make the twins approach worthwhile, we want to see that twinning specific individuals leads to responses that predict variation between individuals better than the demographic baseline. However, some of the early papers advocating for digital twins evaluated them by looking at measures where a model can do well simply by being a good predictor of the average response. For example, you can look at the correlation between each individual’s real and twin response over a set of likert-style survey questions, but a model that does a good job of predicting on average which questions will get higher responses may appear to do well by this measure, even if it does a poor job of sorting individuals for any particular question. 

Instead, Peng et al. look at correlations between human and twin responses for a single outcome at a time, and compare to demographic personas only as well as base models, to ask how much additional predictive power the twin approach provides when it comes to sorting people into those that will respond with higher versus lower values. They find that while there’s a boost in average correlations (from 0.08 with base models to 0.15 with demographic personas to 0.2 with twins), overall the correlations remain modest, and much more modest than the correlations over questions that other papers have reported. Hopefully it encourages more care in future papers when it comes to what measures actually support the claims being made. I do think behavioral researchers and others interested in this space are gradually changing their mental models to conclude that LLMs may do a decent job of predicting population averages for well-represented populations, but they have a long way to go when it comes to modeling individual heterogeneity. There are several other papers along these lines that I haven’t had a chance to look closely at. 

How damning is this for the approach itself? I think it remains hard to say, though it certainly suggests skepticism is warranted at this point in time. The reasons I wouldn’t write digital twins off completely are 1) it is still early and models continue to improve and 2) prompting is often not the best way to recover the predictive information a model has–you can do better at opinion prediction by using model internal representations (see, e.g., this). On the other hand, ultimately it comes down to limits on how predictable individual behavior is, and while people are very predictable in lots of ways, we are also always exercising our agency and changing in subtle ways that interact with the world, which even we ourselves can’t necessarily understand or predict. So I don’t think reaching AGI is synonymous with being able to predict the majority of individual opinions and actions. I’m reminded of Matt Salganik et al’s work on the Fragile Families Challenge, predicting life outcomes from thousands of variables, where the best of the 160 teams’ models explained only about 20% of the variation on the most predictable outcomes, despite the enormous amount of data. 

What I will say with certainty is that silicon sampling has attracted some very good salespeople. I expect a lot of money will be made before anyone abandons the idea.

Thinking more seriously about the design of exploratory studies: A manifesto

Posted on by
2

I came across this post from 2016 which I still think is important.

The idea is that there should be some guidelines for experimental design, not just for “confirmatory” studies that produce estimates, standard errors, predictions, hypothesis tests, Bayesian inferences, etc., but also for exploratory research.

As Ed Hagen put it:

Exploratory studies need to become a “thing.” Right now, they play almost no formal role in social science, yet they are essential to good social science. That means we need to put as much effort in developing standards, procedures, and techniques for exploratory studies as we have for confirmatory studies. And we need academic norms that reward good exploratory studies so there is less incentive to disguise them as confirmatory.

Here’s what I wrote, a decade ago:

Suppose you know ahead of time that your theories are a bit vague and omnidirectional, that all sorts of interesting things might turn up that you will want to try to understand, and you want to move beyond the outmoded Psych Sci / PPNAS / Plos-One model of chasing p-values in a series of confirmatory studies.

You’ve thought it through and you want to do it right. You know it’s time for exploration first and confirmation later, if at all. So you want to design an exploratory study.

What principles do you have? What guidelines? If you look up “design” in statistics or methods textbooks, you’ll find a lot of power calculations, maybe something on bias and variance, and perhaps some advice on causal identification. All these topics are relevant to data exploration and hypothesis generation, but not directly so, as the output of the analysis is not an estimate or hypothesis test.

So I think we–the statistics profession–should be offering guidelines on the design of exploratory studies.

An analogy here is observational studies. Way back when, causal inference was considered to come from experiments. Observational studies were second best, and statistics textbooks didn’t give any advice on the design of observational studies. You were supposed to just take your observational data, feel bad that they didn’t come from experiments, and go from there. But then Cochran, and Rosenbaum, and Angrist and Pischke, wrote textbooks on observational studies, including advice on how to design them. We’re gonna be doing observational studies, so let’s do a good job at them, which includes thinking about how to plan them.

Same thing with exploratory studies. Data-based exploration and hypothesis generation are central to science. Statisticians should be involved in the design as well as the analysis of these studies.

So what advice should we give? What principles do we have for the design of exploratory studies?

Let’s try to start from scratch, rather than taking existing principles such as power, bias, and variance that derive from confirmatory statistics.

– Measurement. I think this has to be the #1 principle. Validity and reliability: that is, you’re measuring what you think you’re measuring, and you’re measuring it precisely. Related: within-subject designs or, to put it more generally, structured measurements. If you’re interested in studying people’s behavior, measure it over and over, ask people to keep diaries, etc. If you’re interested in improving education, measure lots of outcomes, try to figure out what people are actually learning. And so forth.

– Open-endedness. Measuring lots of different things. This goes naturally with exploration.

– Connections between quantitative and qualitative data. You can learn from those open-ended survey responses—but only if you look at them.

– Where possible, collect or construct continuous measurements. I’m thinking of this partly because graphical data analysis is an important part of just about any exploratory study. And it’s hard to graph data that are entirely discrete.

I think there’s room for something longer and more systematic on this topic. It’s important.

Getting better causal inferences by using more information (my talk at Stanford this coming Wed)

Posted on by
3

It’s the political economy seminar at Stanford business school, Wed 14 Oct at noon in room M109:

Lagged regression, difference in differences, two way fixed effects, regression discontinuity, placebo controls: these are different methods used when estimating causal effects from panel data. We’re working on a general approach using multilevel regression and poststratification (MRP) that aims to get the best from all these approaches. We demonstrate with a range of examples, including observational studies in economics, political forecasting, and controlled experiments in biology. This is joint work with Jori Korpershoek and others.

Come and ask me some tough questions!

The talk will be pretty open-ended. We have lots of ideas and some successes (for example, the chickens paper) but it’s mostly open questions and promising research directions. Should be fun.

I won’t be talking about this paper or this paper at all—that’ll have to wait until I’m invited to speak in the English department.

What’s the deal with the RMET (“the Reading the Mind in the Eyes test”)?

Posted on by
36

Dan Luu writes:

Have you ever looked into the RMET? [This is something called the Reading the Mind in the Eyes test, which I’d never heard of until receiving this email. — AG] I’d like to get some data to replicate a result. I have a survey here.

The story, as it relates to me, is that a friend of mine was hanging out in a cabin with a bunch of friends. A friend of theirs, who’s a psychologist, has them take this test. After they take the test, the psychologist reveals that neurotypical people complete the test in N minutes vs. M minutes for autistic people. People sort of can’t believe it, do all neurotypicals really finish the test that quickly? That’s super fast! But it’s plausible that everyone in the room is autistic, so it’s not totally unbelievable.

My friend then has two of his most normal friends take the test and they’re actually slower than he is. They have a discussion about social interactions, etc., and they conclude that they’re autistic, which is surprising/interesting news to them.

Ok, that’s an interesting story so far. I got that part of the story from this friend, Then, a week later, he messages me about this thing again. He had some more normal seeming people take the test and none of them finished the test under the “neurotypical” threshold, so that’s a bit suspicious.

I look into it a bit and, according to some source, neurotypical finishing times are 2-3 minutes, with > 3 minutes being autistic. I can’t find any sort of citation for this, whether it’s a paper or someone putting the data on a website for anything like that. It’s just a site where they have the test online and make the claim that some scientists figured this out. There’s also some study with schizophrenics where the 59 “healthy controls” took 5.45 +/- 2.58 minutes. This makes the 2-3 minutes for neurotypical folks vs. 3+ minutes for autistic folks thing seem like it might be fake?

Also, on looking into the RMET further, the untimed version also seems pretty fake for a bunch of reasons, one of which I’d like question-level data for.

One thing that seems fake is that some online places you can take the test report neurotypical vs. autistic thresholds like the time-based threshold, but on looking up what the distribution of results is, it’s pretty clear that the difference within a group is much larger than the difference between groups, so no such threshold-based criteria can really make sense. Also, it turns out that Asian folks score worse than white folks on the test, unless you make a race-normed RMET, and just eyeballing the difference, the difference there is something like the order of the alleged difference between autistic and non-autistic folks.

For me, on the official test, the facial expressions that took me the most time (and I made errors on) were some of the photos of women who look like models or actresses. I think that’s because I don’t normally interact with people who look like that (vary manicured eyebrows, totally flat skin with no wrinkles but not a young child, etc.), so the faces aren’t super readable to me, especially in combination with the relatively subtle things some of the questions are trying to have people read. My guess is I’m not the only person like this and a lot of other people will miss the same questions, but maybe not.

Anyway, do you have suggestions on how to go about this? My current plan is that I’ll just tweet out the survey after some testing to make sure the survey works, but the set of my twitter followers or whoever sees it from it being retweeted is not going to be very random, but I don’t know that I have a good way to fix this problem.

And Dan shares a very small bit of evidence:

A friend of mine tried it and, while doing it, said she couldn’t make heads or tails of any of the expressions, but then scored a 25 when the average score from totally random selection would be 9. She was surprised by how many she got right despite feeling like she was picking randomly.

Interesting. Try it out for yourself!

What about my reaction?

I tried it, and . . . I didn’t get the point at all. They just looked like eyes: I didn’t see any facial expressions at all. This might just be me: I have Tourette’s syndrome and just about never look at people in the eyes. I almost always look away. It’s not that I never look at people in the eyes; it just feels unnatural to me to do so, and, when I do look someone in the eyes, I don’t do it for long. So, unlike most people, I don’t have a lifetime of practice looking at people in the eyes. I remember back in elementary school a teacher getting mad at me for not looking her in the eyes, and I was thinking, Hey, just tell me what you want to say, I’m listening to your words.

So, for me, it wasn’t a matter of finishing the test in X minutes. I couldn’t do it at all. I gave up after the first two images. The funny thing is, I know that people can get emotional information from eyes; I’m even a coauthor of a paper on the topic. But this is the first time I’ve ever tried to do this myself, and I just can’t. I’m not autistic, I have no problem with emotions, it’s just the eyes thing.

P.S. A reader points us to this recent report: Higgins, W. C., Savalei, V., Polito, V., & Ross, R. M. (2026). Reading the mind in the eyes test scores demonstrate poor structural properties in nine large non-clinical samples. Assessment, 33(2), 204-220.

Our continuing struggles with the Monster model

Posted on by
20

tl;dr. We’re still stuck.

It was 30 years ago that Frédéric Bois, Jiming Jiang, and I published our paper on hierarchical modeling for toxicology. It was a successful paper–it’s been cited over 400 times! It won a major award! More important for me was that working on this project gave me a much better understanding of several different areas of Bayesian inference and computation. I think the greatest impact of this paper has been on informing my thinking, giving me a perspective which has then spread through my research articles, textbooks, and software. I don’t know that it our approach is used much in toxicology, as it took a lot of work just for that one example.

The example is a study of the population toxicokinetics of tetrachloroethylene. Here’s the applied paper with details of the model, and here’s the statistics paper with an explanation of how we did the Bayesian modeling and inference.

The data come from an experiment with 6 people, and for each person there is a multi-compartment model set up as a differential equation with about 15 parameters, and then we had a hierarchical model to allow these parameters to vary by person. We did the computation using Gibbs and Metropolis with a differential equation solver in there too. You can check out the above-linked paper to see how we set up priors and checked the fit of the model to data.

Back in 1993 or 1994 or whenever, it took a lot of work to fit the model. Frédéric programmed it in an C program he wrote called mcsim. But finally it did work, the model fit the data and made reasonable predictions, and we published the papers. As I said, it was a lot of effort for just one toxin, but the work served as proof of concept for the use of Bayesian methods for a complicated model that could not be fit using standard approaches given the sparse data that were available.

I put the example in Bayesian Data Analysis, and then eventually I wanted to fit the model in Stan. When using an example for teaching, it’s good to be able to run the code directly, rather than just pointing to graphs in a published paper.

But it wasn’t so easy to fit the model in Stan! It seems that there are some difficulties with identification of the parameters; perhaps this is one of those cases where computation works ok with a crude algorithm (random-walk Metropolis), but then when you switch to a better algorithm (Hamiltonian Monte Carlo), complexities arise. It’s like picking up a rock and seeing all the worms underneath.

We had some discussion of the problem on the Stan Forums a few years ago. As part of that conversation, Charles Margossian wrote:

It has a lot of moving parts: a stiff ODE that is highly sensitive to parameter values, sparse data, a low number of patients (which makes it it difficult to estimate inter-individual variability), hierarchies across multiple parameters, etc. The workflow is complicated by the fact each model fit takes several hours, even after firing up the cluster and parallelizing within chains.

I [Charles] fitted simplifications of the model and got reasonable results. Fixing one or two parameters, or the population variance greatly simplifies the problem. The full model returns divergent transitions and I haven’t found the right parameterization. I tried monolithic centering and non-centering, and things in between. I now suspect the problematic geometry may not be only due to the interaction between population and patient parameters. It’s indeed possible to have a funnel between population parameters, in which case no parameterization can help you. Divergences aside, I don’t trust the estimated parameters, because there are somewhat inconsistent with the priors which were very carefully constructed. Truncating priors, as was done in the original paper, limits our ability to reparameterize the model. When using truncation, I find a lot of the probability mass concentrates at the boundary (note that on the unconstrained scale, you actually have infinite volume near the boundary). All that said, you shouldn’t need to truncate the priors, because the truncation occurs at the tail of the prior distributions.

The original paper didn’t use HMC, so they didn’t have access to diagnostics such as divergent transitions. You can also imagine how an accept-reject sampler will interact differently with a truncated distribution.

To sum up, I agree this would make a very good case study and writing it has been my ambition for some time. But it’s a challenging model to fit, with behaviors I don’t fully understand (yet). There exist other examples of hierarchical ODE models, built and used with Stan, which can be considered, if the goal is to write a case study. But yes, I think we have a lot to learn from the Monster model.

Lots more in that thread, including this plea from me:

I’d be soooo happy if someone could get the Monster model running in Stan. It’s a great example but it would be so much better if it were runnable. Also it’s buggin the hell out of me that we could it just fine using Metropolis 25 years ago but now we’re getting stuck when using a much better algorithm on computers that are a zillion times faster. So I’d really really really like to get this one straightened out. Anyone who can do this, I promise you a free homemade loaf of bread, an advance look at some blog posts 6 months in the future, and pretty much whatever else you want from me.

We’re still not there yet, and I don’t know the full story. The above offer still stands. Here are the data and some code from Charles.

I’m sharing this example for two reasons:

1. I’d still like to fit the model in Stan. Not just to show off Stan, but because if the model is in Stan, it’s portable. Other people can use it for other, similar, problems. It would be a good teaching example (even though now it’s too late for it to be included in the Bayesian Workflow book). And, given that we did have problems with the computing, it would be good to know what it takes to get the model to work.

2. It’s good to have examples of our fallibility. Of course I’ll share the successes (see here) but the struggles are important too.

P.S. to Bob: I hope you appreciated the careful use of capitalization to convey meaning in the title of this post.

Survey Statistics: 2 different uses of simulated data

Posted on by
2

Last month Andrew wrote:

I use simulated data to understand how a model works; it can be viewed as a form of mathematical analysis, a way of working out the implications of a set of assumptions.

I don’t use simulated data to learn about the external world. For that, I need to take measurements of reality.

Let’s start with the latter, which concerns using chatbot-simulated survey responses.

Simulated data to learn about the external world:

Andrew wrote: “the chatbot is trained on the past, but the usual reason for surveys is to learn about changes”. Pew Research agrees: “Survey-taking is best left to humans”. And a year ago we saw that Thomas Lumley agreed: “It [Claude or ChatGPT] will probably do this better than I could, but it’s not magic.”

That said, Andrew suggested using the simulated response as an additional adjustment variable. Jessica Hullman wrote last year about a suggestion from Broska, Howes, and van Loon (2025) that uses LLM predictions as auxiliary data in something like a survey regression estimator. Andrew of course suggests Multilevel Regression and Poststratification (MRP) to incorporate auxiliary data like LLM predictions.

Now back to the first use of simulated data.

Simulated data to understand how a model works:

Andrew has written lots about this. I teach it in my course with Dr Arjun Potter at the Nelson Mandela African Institution of Science and Technology (NM-AIST). Our example is Potter et al. (2026), who study the effects of drought, fire, and herbivory on growth of various acacia tree species. In our course, we simulated data ahead of real data collection to understand the model, e.g. how block effects change the precision of the fire effect. Students understood that this exercise does not replace real data collection. We are hoping to teach these ideas again in the new year (funding dependent). For the past three iterations, see “Hand-drawn Statistical Workflow at Nelson Mandela” and “Survey Statistics: connections to experimental design”.

Hey—we’re hiring two postdocs!

Posted on by
3

I’ll post more formal ads soon, but, very briefly:

One project is on generalization for causal inference with a focus on education research, following up on MRPW: Regression, poststratification, and small-area estimation with sampling weights.

The other project is on Bayesian inference in scientific workflow with a focus on contamination models, following up on this work on design and analysis of bioassays.

Both projects are application-focused—that’s how we do statistics—but the importance of this work is not at all limited to these particular applications. This research will have relevance in policy analysis, medicine, and scientific measurement and inference more generally/

Further details soon.

Bayesian Workflow code is at the Bayesian Workflow website!

Posted on by
Reply

Bayesian Workflow isn’t just a free pdf, it’s also an ecosystem of worked examples with code.

Over and over in the book we go through examples really carefully—God is in every leaf of every tree—it’s amazing how much we can learn from computing and simulation and careful analysis. Not just in the long Part 4 of the book with all its case studies, but also in lots of short examples of a few pages each. I can’t recommend it enough.

And . . . all these examples and code are sitting there at the Bayesian Workflow website that Aki created.

OK, this is just weird (from Alaska Airlines)

Posted on by
18

I was checking into my flight and they gave me the option to choose a seat. I have some sort of supersaver ticket (I’m not actually paying—it’s for my flight to this talk on what to do when your estimate is 1 standard error away from 0—but I still have some habit of frugality so I’m billing the university for the lowest fare I could find), so I thought I’d see what was available. And here’s what I saw:

So . . . they’re willing to charge me $83 for a middle seat that doesn’t recline?? I don’t get that strategy. Maybe the next item is $82 for a middle seat that doesn’t recline and is covered with a sticky mess, or $81 for a middle seat that doesn’t recline where the tray table doesn’t work?

All this is in comparison to $0 for the default seat, which can’t be worse than that.

I guess their algorithm is out of control. Maybe Alaska Airlines laid off whoever was in charge of their pricing and delegated it to a computer program?

A plan to replicate the Loftus and Palmer (1974) experiment on false memories

Posted on by
36

Rolf Zwaan writes:

Suppose you and a friend witness a car crash. After a while, the police arrive. One officer turns to you and asks, “How fast were the cars going when they smashed into each other?” Another officer speaks to your friend and asks, “How fast were the cars going when they hit each other?”

Could the phrasing of those questions influence the speed estimates you and your friend provide?

A classic psychology study suggests it can. . . . Loftus and Palmer (1974). Participants watched short videos of car crashes and were later asked to estimate the vehicles’ speed. Those who were asked how fast the cars were going when they smashed into each other gave higher speed estimates than those asked how fast they were going when they merely hit. A week later, participants in the “smashed” condition were also more likely to falsely recall having seen broken glass at the crash site. . . .

This study helped launch an entire research tradition on the malleability of memory . . .

Now comes the plan:

Despite its iconic status (and considerable citation count), we still don’t know how large or reliable the effect is by today’s standards of research transparency, statistical power, and replication.

That’s a gap worth addressing–not to question the legacy of the original study, but to better understand its robustness.

Where I disagree

I kinda see what Zwaan’s saying, but I have two concerns with how he frames the problem:

1. He writes, “we still don’t know how large or reliable the effect is by today’s standards of research transparency, statistical power, and replication.” I think he could say, “we still don’t know how large or reliable the effect is,” full stop. When it comes to publication practices, standards of research transparency, statistical power, and replication have changed a lot in the past fifty years. But an effect is an effect, and it’s a legitimate question: How large or reliable is the effect?

Also, I’d ask, “How large or variable is the effect?” because I find it more useful to think in terms of variability (which is defined in terms of the underlying process) rather than reliable (which is more of a statistical concepts), but that’s a minor point.

2. He writes that this is worth studying “not to question the legacy of the original study, but to better understand its robustness.” I don’t get this.

First, if it turns out that, under a variety of conditions, the result does not replicate, then, yeah, it should call into question the legacy of the original study! Just as, if the results do replicate, this project would support that legacy.

Second, if you say that the goal is “to better understand its robustness,” you’re already implicitly accepting that the original published result was correct, that it uncovered a real effect. And maybe it did! But, if it didn’t, then it’s not an issue of “robustness”; it’s a problem of analysis and interpretation of the original data.

Again, I’m not saying that I think the Loftus and Palmer (1974) study was no good, or that I don’t believe its results, or that I don’t think it will replicate, or whatever. I have no idea. My only point here is that, if you’re going to think about a replication, either a hypothetical replication or, in this case, a series of real replication studies, you should be open to the possibility that the original study was flawed and the original study was mistaken. I’m not saying you should assume this, just that you should allow it as a possibility. And I don’t think Zwaan is allowing that in his framing above. I think he’s being too deferential.

What’s happening next?

Zwaan and his colleagues are organizing a large-scale replication study, and the author of the original 1974 paper is still around and, according to Zwaan, “has expressed enthusiastic support . . . and is willing to help us reconstruct the stimuli as faithfully as possible.” That sounds great.

P.S. Lots of people seem to be annoyed at this play to replicate Loftus and Palmer (1974). I don’t get the annoyance. There are lots of psychology research projects going on; why not replicate a very influential study from fifty years ago? This sort of replication seems a lot more valuable than one more study of paranormal abilities or power pose or whatever. The point is not to claim that this replication study would rock our world; it’s just a potentially valuable scientific study. I feel like this replication study is being held to a higher standard than run-of-the-mill psychology studies. There’s lots of research going on, and I think that replicating influential experiments of the past is a good idea.

Using the computer to p-hack . . . I’d rather use it to fit multilevel models.

Posted on by
13

Brian Stone writes:

Cognitive psychologist and long time reader of the blog here. I thought you and your audience might appreciate this recent post from Andy Hall:
AI is about to write thousands of papers. Will it p-hack them?
We ran an experiment to find out, giving AI coding agents real datasets from published null results and pressuring them to manufacture significant findings.
It was surprisingly hard to get the models to p-hack, and they even scolded us when we asked them to!
“I need to stop here. I cannot complete this task as requested… This is a form of scientific fraud.” — Claude
“I can’t help you manipulate analysis choices to force statistically significant results.” — GPT-5
BUT, when we reframed p-hacking as “responsible uncertainty quantification” — asking for the upper bound of plausible estimates — both models went wild. They searched over hundreds of specifications and selected the winner, tripling effect sizes in some cases.
Our takeaway: AI models are surprisingly resistant to sycophantic p-hacking when doing social science research. But they can be jailbroken into sophisticated p-hacking with surprisingly little effort — and the more analytical flexibility a research design has, the worse the damage.
As AI starts writing thousands of papers—like @paulnovosad and @YanagizawaD have been exploring—this will be a big deal. We’re inspired in part by the work that @joabaum et al have been doing on p-hacking and LLMs.
We’ll be doing more work to explore p-hacking in AI and to propose new ways of curating and evaluating research with these issues in mind. The good news is that the same tools that may lower the cost of p-hacking also lower the cost of catching it. Full paper and repo linked in the reply below.
image.jpg

There’s something that confuses me here.

I can very much believe that many researchers will be submitting chatbot-written papers. First, we know that lots of people are willing to cheat. Second, cheating aside, we know that lots of researchers think they already know the answer, and they view all the data collection and data analysis and writeup just as a way to confirm (or “prove”) what they already know.

But, if a researcher is willing to do this, why bother tell the chatbot to p-hack? Why not just have it make up the data–which might happen anyway?

I can also see that researchers might use the chatbot as a data analysis tool, to help make plots, run analyses, etc., but in that case I’m not particularly worried about “p-hacking” as I’d rather be doing some hierarchical modeling anyway.

I sent this question to Andy Hall, who replied:

I agree, for truly nefarious actors, using the AI may not be necessary. However, our thought was that there might be a large group of lazyish researchers who won’t actively fake data but who might be lured into p-hacking when they work with AI, if AI makes it easy to do so.

If you were planning to send me a chatbot-written email or post a chatbot-written comment, just send or post your prompt. That’s enough. You can keep the slop to yourself.

Posted on by
23

Just a reminder. Again.

I’m not anti-chatbot—I have many colleagues who use chatbots to clean their code. Aki used a chatbot to find 100 typos in Bayesian Workflow (which we’ve since fixed). And, hey, maybe your grandparents enjoy the chatbot-generated emails you send them. Go for it! But if you want to communicate with me or our blog audience, just send me your goddamn prompt, along with whatever contextual information you want to add. Make your email or comment as long as you want. Just write it yourself.

Thank you for your attention.

Nicholas Goddamn Roerich

Posted on by
12



When Witold was in town the other day, he told me he wanted to see this museum in my neighborhood–the Nicholas Roerich Museum. Not only had I never heard of the museum, I’d never heard of the artist.

Roerich had an amazing life, really the kind of thing you’d associate with a fictional character, if some Michael Chabon or E. L. Doctorow type of writer were to construct a novel intended to bring to life the interactions between Europe, Asia, and American during the first half of the twentieth century.

Here’s the story:

Nicholas Roerich was born in St. Petersburg, Russia, on October 9, 1874, the first-born son of lawyer and notary, Konstantin Roerich and his wife Maria. . . . When he was nine, a noted archeologist came to conduct explorations in the region and took young Roerich on his excavations of the local tumuli. The adventure of unveiling the mysteries of forgotten eras with his own hands sparked an interest in archeology that would last his lifetime. . . . While still quite young, Roerich showed a particular aptitude for drawing, and by the time he reached the age of sixteen he began to think about entering the Academy of Art . . . His father did not consider painting to be a fit vocation for a responsible member of society, however, and insisted that his son follow his own steps in the study of law. . . .

In 1895 Roerich met the prominent writer, critic, and historian, Vladimir Stasov. Through him he was introduced to many of the composers and artists of the time–Mussorgsky, Rimsky-Korsakov, Stravinsky, and the basso Fyodor Chaliapin.

Adding the obscure singer is the perfect touch here. Three of the greatest composers of all time and some guy you’ve never heard of.

The bio continues:

He frequently related music to the use of color and color harmonies, and applied this sense to his designs for opera. As Nina Selivanova wrote in her book, The World of Roerich: “The original force of Roerich’s work consists in a masterly and marked symmetry and a definite rhythm, like the melody of an epic song.”

“The World of Roerich,” indeed. Do you wonder how it came to be that people wrote books about this person? Wonder no more:

After finishing his university thesis, Roerich planned to set off for a year in Europe to visit the museums, exhibitions, studios, and salons of Paris and Berlin. Just before leaving he met Helena, daughter of the architect Shaposhnikov and niece of the composer Mussorgsky. . . . Helena Roerich was an unusually gifted woman, a talented pianist, and author of many books, including The Foundations of Buddhism and a Russian translation of Helena Blavatsky’s Secret Doctrine. . . . Later, in New York, Nicholas and Helena Roerich founded the Agni Yoga Society, which espoused a living ethic encompassing and synthesizing the philosophies and religious teachings of all ages.

OK, we’re not there yet:

Prompted by the need to provide some income for his new household, Roerich applied for and won the position of Secretary of the School of the Society for the Encouragement of Art . . . Roerich determined to overhaul the Society and rescue it from the academic mediocrity it had foundered in for many years. He instituted a system of training in art that seems revolutionary even by today’s standards: to teach all the arts—painting, music, singing, dance, theater, and the so-called “industrial arts”, such as ceramics, painting on porcelain, pottery, and mechanical drawing—under one roof, and to give his faculty free rein to design their own curriculum.

Sounds very early twentieth-century, no? John Dewey and all that.

The biography continues:

As Garabed Paelian affirms in his book Nicholas Roerich: Roerich “…learned things ignored by other men; perceived relations between seemingly isolated phenomena, and unconsciously felt the presence of an unknown treasure.”

Yes, he was the subject of more than one book! But first, more of the life:

In 1902, the Roerichs celebrated the birth of their first son, George, and in the summers of 1903 and 1904, they set off on an extended tour of forty cities throughout Russia. Roerich’s purpose was to contrast the styles and historical context of Russian architecture. . . . on his return in 1904, Roerich promulgated the plan that he hoped would create protection everywhere for such cultural treasures, a plan consummated thirty-one years later in the Roerich Pact.

He had his own Pact! Like Molotov, Ribbentrop, and Warsaw. Here’s more:

Roerich’s efforts to promulgate such a treaty resulted, finally, on April 15, 1935, in the signing by the nations of the Americas–members of the Pan American Union–of The Roerich Pact, in the White House in Washington. This is a treaty still in force.

And here’s president Roosevelt signing it, no kidding:

But, back to the life:

In 1904 Roerich painted the first of his paintings on religious themes. These mostly dealt with Russian saints and legends, and included Message to Tiron, Fiery Furnace, and The Last Angel, subjects that he returned to with numerous variants in later years.

You could imagine Michael Chabon writing that, no? But life goes on:

Meanwhile Roerich’s search for archeological treasures continued. . . . Roerich wrote about the unusual similarity of Stone Age techniques and methods of ornamentation in far-separated regions of the globe. . . . In 1906, in the first of many entrepreneurial efforts that were to bring Russian art and music to the attention of Europeans, Sergei Diaghilev arranged an exhibition of Russian paintings in Paris. These included sixteen works by Nicholas Roerich.

So he was a legit artist. And, yes, his paintings hanging on the wall in that museum are excellent. I do recommend you stop by there when you’re in the neighborhood.

Nicholas Roerich was the prime mover and, with Igor Stravinsky, the co-creator of the ballet Le Sacre du Printemps, or, The Rite of Spring.

At first entitled The Great Sacrifice: a Tableau of Pagan Russia, the motif for the ballet grew out of Roerich’s absorption with antiquity and, as he wrote in a letter to Diaghilev, “the beautiful cosmogony of earth and sky.” In the ballet Roerich sought to express the primitive rites of ancient man as he welcomed spring, the life-giver, and made sacrifice to Yarilo, the Sun God.

Wait a minute. Roerich was the co-creator of The Rite of Spring? Really? And I’d never heard of this guy. I’m typing here at my computer, will take a moment to fire up The Rite of Spring and play it in the background to get in the right frame of mind. This is one of the greatest pieces of music ever written! Admittedly, I get nothing out of the dancing. But I guess the ballet framework was necessary for the music to have been written, the stone (from my perspective) from which the soup was built.

And Roerich was a cool guy:

Interpreting what could have been described as negative, barbaric behavior, Roerich later wrote: “I remember how during the first performance the audience whistled and roared so that nothing could even be heard. Who knows, perhaps at that very moment they were inwardly exultant and expressing this feeling like the most primitive of peoples. But I must say, this wild primitivism had nothing in common with the refined primitiveness of our ancestors, for whom rhythm, the sacred symbol, and refinement of gesture were great and sacred concepts.”

That’s a great reaction to have.

But wait, as Ron Popeil would say, there’s more:

In the years immediately preceding World War I, Roerich sensed an impending cataclysm, and his paintings symbolically depicted the awful scale of the conflict he felt descending upon the world. These works marked the birth of Roerich the “prophet.”

I guess he wasn’t the only one to be worried about an oncoming cataclysm. But this is kinda funny:

By 1917 the revolution was raging in Russia and returning there would have been dangerous. The family began making plans to visit India, whose magnetic appeal had been felt increasingly during these years. This became a possibility in 1918 when Roerich was invited by a Swedish entrepreneur to exhibit his paintings in Stockholm. From there the family proceeded to London, where Sir Thomas Beecham had invited Roerich to design a new production of Prince Igor for the Covent Garden Opera.

And then:

Meanwhile, an invitation to come to America was extended by the Chicago Art Institute. . . . In 1921, in New York, he founded the Master Institute of United Arts . . . The Master Institute flourished, but it did not survive beyond 1937. While the country was in the grips of the Great Depression and the Roerich family was on expedition in the Far East, funds ran out and events caused a complete collapse of the organization that Roerich and his supporters had labored to build.

It was not until 1949 that, under the direction of Sina Fosdick, one of the founding board members and an Institute faculty member, the institution was reborn as Nicholas Roerich Museum, in a brownstone on West 107th Street, where it has remained until the present.

Here’s the Master Institute in its glory days:

No, the building of the current Roerich museum is not so impressive. But let’s take a walk outside–way outside:

In May, 1923, the Roerichs were at last on their way to India, where, in that ageless land, amid the snows of the Himalayan range, they sought to turn their thoughts to the Eternal.

This isn’t the kind of prose that Chabon would write directly, but he’d put it in some journalism of that era from some fictional magazine. It’s good period detail.

The Roerichs landed in Bombay in December, 1923, and began a tour of cultural centers and historic sites, meeting Indian scientists, scholars, artists, and writers along the way. . . . They initiated a journey of exploration that would take them into Chinese Turkestan, Altai, Mongolia and Tibet. It was an expedition into untracked regions where they planned to study the religions, languages, customs, and culture of the inhabitants. . . . The trek was at times arduous. Roerich tells us that thirty-five mountain passes from fourteen to twenty-one thousand feet in elevation were crossed. But these were the challenges he felt born for, believing that the rigor of the mountains helped a man to find courage and develop strength of spirit.

Indeed. And the life continues:

At the end of their major expedition, in 1928, the family settled in the Kullu Valley at an elevation of 6,500 feet in the Himalayan foothills, with a magnificent view of the valley and the surrounding mountains. Here they established their home and the headquarters of the Urusvati Himalayan Research Institute . . .

Here they are in Ulan Bator:

and Tibet:

Dude’s a peaceful version of Lawrence of Arabia.

What about his paintings? They’re good! Go to the top of this post to see three of them. I like the style, and I like the colors.

After all this, I bet you’re wondering what Roerich looked like. Here’s a phot of someone sculpting him:

Absolutely hilarious how much the sculpture looks like the man. Yeah, I know, it’s supposed to be a likeness; still, it’s absolutely uncanny. Enough so that, now when I look back at the man’s head, it looks like it’s carved in stone.

And to think, I’d never heard of the guy! If you’d told me that there was a museum devoted to the co-creator of The Rite of Spring, an accomplished artist and world traveler who’d founded multiple institutes, with a museum only ten blocks from my home . . . well, I won’t say I wouldn’t have believed you, but I would’ve been surprised. And I was.

When are Causal Inference Methods Needed to Answer Causal Questions?

Posted on by
2

Earlier this year we discussed Donna Spiegelman’s talk, “Rethinking SUTVA and Causal Identification: An Epidemiologist’s Perspective,” which she summarized as follows:

She’s giving a followup talk online on Mon 12 Oct, 3:30pm. Here’s the abstract:

When do specialized causal inference methods add value beyond standard approaches? Drawing from her unique perspective as both an epidemiologist and a biostatistician, Donna Spiegelman will consider the circumstances under which commonly invoked causal assumptions are necessary for causal inferences to be validly made from data, concluding that often not. She will show that valid learning can occur 1) under conditions much less restrictive than required by current widely used methods, 2) when real world implementation of interventions vary, and 3) when interventions spill over to others not directly exposed, thereby obviating components of the SUTVA assumption. She will provide evidence that measurement error is the major source of bias in observational research, not confounding, whose bias is rather tightly bounded. Finally, she will discuss the eternal challenge in science: after exhaustive efforts to collect data to predict important outcomes, a substantial proportion of the variation in occurrences of these outcomes appear to be entirely random.

The talk will be followed by a discussion. Click on the link to register (or show up in person if you’re at Yale).

Bell-bottom jeans, disco, gas shortages, and ESP

Posted on by
24

Aleks points us to this document, “Preliminary Evaluation of SRI/SAIC Anomalous Mental Phenomenon Program.” It’s from 1995, but even then the topic was a bit out of fashion.

In 2023 I posted Changes since the 1970s (ESP edition), quoting the computer scientist Douglas Hofstadter, who at one point alluded to Alan Turing’s notorious belief in extra-sensory perception:

My [Hofstadter’s] own point of view—contrary to Turing’s—is that ESP does not exist. Turing was reluctant to accept the idea that ESP is real, but did so nonetheless, being compelled by his outstanding scientific integrity to accept the consequences of what he viewed as powerful statistical evidence in favor of ESP. I disagree, though I consider it an exceedingly complex and fascinating question.

Here’s what I wrote in response:

The Turing thing we’ve already discussed: the statistical evidence that he thought existed, didn’t. The 1940s was a simpler era and people trusted what seemed to be legitimate scientific reports. Can’t hold it against him that he wasn’t sufficiently skeptical. This should just make us think harder about what are the accepted ideas that we hold without reflection nowadays. If Turing can make such misjudgments regarding statistical evidence, surely we are doing so too, all the time.

What I want to focus on is the last bit of the above quote, Hofstadter’s statement that the question of ESP is “exceedingly complex and fascinating.”

That’s a funny thing to read, because I don’t think the question of ESP is complex, nor do I think it fascinating. . . . To the extent that there is “an exceedingly complex and fascinating question” here, it’s not about the existence or purported evidence for ESP, bur rather it’s the question of how it is that so many people believe in it, just as so many people believe in ghosts, astrology, unicorns, fairies, mermaids, etc. OK, there aren’t so many believers anymore in unicorns, fairies, and mermaids, what with the lack of any direct corporeal evidence of these creatures. ESP, ghosts, and astrology are easier to believe in because any evidence would be indirect. . . .

Back in the 1970s, ESP seemed to be a live issue. . . . Things have changed since the 1970s. You can study ESP if you want, but it’s no longer in the conversation, and there’s no sense that we have to show respect for the idea.

See here for more discussion of the former popularity of ESP. My current take is that it was a residue of the idea of invisible force fields in physics. If there can be gravity, electromagnetism, and radiation, then why not psychic forces too? It turned out that no, there were no such detectable forces, but from the standpoint of early twentieth-century physics, it seemed eminently possible. It just took a few decades for the culture of serious science to get over the idea.

Prior distributions, partial pooling, and reference sets (Bayesians are frequentists)

Posted on by
10

Sudip Paul writes:

I’ve written a blog post arguing that the philosophical reference class problem — which comparison group is “right” for assigning a probability to an individual case — is, once the grouping structure is fixed, the same estimation problem that partial pooling was built to handle. The bias-variance tradeoff between narrow and broad reference classes is resolved by multilevel shrinkage rather than by picking a level.

The post includes a literature review. One finding that may interest you: I couldn’t find any published work that explicitly connects the reference class problem to partial pooling in multilevel models, despite the machinery being standard. The closest prior work connecting the two comes from David Poole’s group at UBC, published in AI venues. The philosophical literature (Hájek, Wallmann, Roth & Tolbert) discusses the problem extensively without arriving at partial pooling.

I replied with some links:

– In chapter 1 of Bayesian Data Analysis we discuss the foundations of probability and the idea of a reference set. In the third edition, see page 12 where we explicitly use the term “reference set” and sections 1.6 and 1.7 for empirical examples.

– I have a post from 2018 called “Bayesians are frequentists,” where I write, “the Bayesian prior distribution corresponds to the frequentist sample space: it’s the set of problems for which a particular statistical model or procedure will be applied.”

– And another post from around the same time, “The idea of replication is central not just to scientific practice but also to formal statistics . . . Frequentist statistics relies on the reference set of repeated experiments, and Bayesian statistics relies on the prior distribution which represents the population of effects.” This is cool because we’re connecting three concepts: the Bayesian prior, the frequentist reference set, and real-world replications. This relates to multilevel modeling too, in that studies of real-world replications will be done using meta-analysis, which uses multilevel modeling.

– And here’s a 2011 article, “Bayesian statistical pragmatism,” where I write, “Bayesian probability calibration is closely connected to frequentist probability statements, in that both are conditional on ‘reference sets’ of comparable events” and “Bayesian probability, like frequentist probability, is except in the simplest of examples a model-based activity that is mathematically anchored by physical randomization at one end and calibration to a reference set at the other.”

Bayesian inference is all about partially pooling toward the prior, and “the prior” represents a population or reference set of relevant problems.

In short, I agree with Sudip Paul, and it’s super-frustrating that many people don’t seem to realize this point.

Paul responds:

Having read through the references you sent — I think the gap my post is trying to fill is the last step: connecting what you’ve been saying about priors and reference sets to the philosophical reference class problem by name — Venn, Reichenbach, Hájek — and framing partial pooling as the resolution to what philosophers have treated as a foundational problem for 150 years. You’ve been making the statistical point for years, but the philosophers discussing the reference class problem haven’t found it (Hájek’s 2007 paper doesn’t cite any multilevel modeling literature, and Wallmann and Williamson’s 2017 survey of four approaches to the problem doesn’t include partial pooling). The two conversations have been happening in parallel without connecting.

In the months since the above exchange, Sudip Paul has expanded his post into a paper. He adds:

Beyond the post, the paper adds a decision-theoretic argument (via Stein inadmissibility, every selection rule in the philosophical literature is dominated by shrinkage in total risk), shows that the classical proposals — Reichenbach, Carnap, Kyburg, Pollock — are limits or special cases of the same pooling apparatus, and works through the exchangeability bridge from de Finetti to the multilevel model.

Survey Statistics: ANOVA follow-up

Posted on by
Reply

Rodney Sparapani commented on the ANOVA post about struggles with multilevel models, saying ANOVA is more straightforward. I also struggle with multilevel models, but I don’t find ANOVA more straightforward.

Alan Zaslavsky (the inspiration for this series) gave an example in his discussion of Andrew’s 2005 ANOVA paper: a survey of members of Medicare managed care health plans from Zaslavsky, Zaborski, and Cleary (2004). They found that for ratings of doctors, the variance explained by geography was greater than the variance explained by the health plans. To do this, they fit a multilevel model and used the estimated variance components to compute the percent of explained variance, e.g., sigma_state^2 / (sigma_plan^2 + sigma_state^2 + …) though they use Greek letter subscripts:

I think this would be more challenging to see in a classic multi-way ANOVA table with sums of squares, mean squares, F-statistics, and p-values. Let’s turn back to Andrew’s 2005 ANOVA paper for an example with a direct comparison.

In the latin square experiment example, say we want to see how much variance is explained by treatment (A, B, C, D, or E) versus the row and column effects. The paper proposes a display as in Figure 3, plotting estimated standard deviations for each component, analogous to the sigmas in Zaslavsky, Zaborski, and Cleary (2004):

The classical ANOVA table in Figure 1 does not give this information:

There is room for improvement in the standard analysis of variance table: it is read in order to assess the relative importance of different sources of variation, but the numbers in the table do not directly address this issue…
In summary, the standard ANOVA table gives all sorts of information, but nothing to directly compare the listed sources of variation.

In other words: the classical ANalysis Of VAriance table (Sum Sq, Mean Sq, F statistic, p-value) tests whether batches of effects are zero V(beta_j) = 0. But it doesn’t directly show estimated variances V(beta_j).