This is Jessica. I’ve been posting recently on questions related to individual probability, i.e., assigning probability to individual events, related to a course I just wrapped up where this was a theme. For example, previously we talked about how statistical exchangeability–where the joint distribution of a random variable is unaffected by the order in which it’s observed–is from one perspective all you need to reconcile differences between “groupist” (e.g., probability as long run frequency over a set of similar events) and “individualist” notions of the probability of a single event (e.g., a probability as based on some expert’s beliefs about a specific event conditioned on some prior experience). A fundamental problem with individual probabilities is the reference class problem: to assign a probability to an event that will only happen once, we have to identify some group of events that we believe capture the essential characteristics of the event in question and estimate the probability over that. But often there will be several equally-appropriate-seeming reference groups that we must choose between. For example, if we have a criminal defendant with a combination of prior arrests and convinctions that we have never seen together before, which subset of these features do we use in estimating their probability of commiting another crime if released, assuming that using different reference groups result in different conditional probabilities? 

The reference class problem is associated with the idea that there is often no unique way to assign a conditional probability to a particular event. Consequently, predictive multiplicity–the fact that machine learning problems often admit multiple competing models that perform more or less equally well–has been accepted as a consequence of the underspecification of individual probabilities in the machine learning literature. Multiplicity is sometimes described as the Rashomon effect, or through the idea of a Rashomon set, i.e., a set of models that predict equally accurately but which make conflicting predictions on some subset of data space. For example, in Breiman’s well known paper on the two cultures of statistical modeling, he uses the Rashomon effect to argue that we have to be careful not to draw conclusions about the process that generated some data using a single explanatory model unless we can somehow rule out all the competing models. This possibility has been discussed more recently in conjuction to concerns like algorithmic fairness, where the presence of other equally accurate models that assign some specific person the opposite prediction is considered unsettling. 

But is multiplicity due to the underspecification of individual probabilities really a fundamental property of predictive models learned from data? Imagine you have two models that are equally supported by the data (and make predictions close to the true conditional probabilities for the various possible reference groups). But they disagree non-trivially in their predictions. Should we accept that such situations exist and cannot necessarily be resolved? 

 A new paper by Roth and Tolbert reframes this as “the reference class problem at scale.” More concretely, say you have some distribution over a collection of elements representing different combinations of features, which might represent, e.g., different people’s records. Assume some true function F that maps from features of these records to outcomes. We can define a set of reference classes representing subsets of the elements; for example, if we have a universe of elements representing different combinations of an age, race, and income variable, then we can define subsets for different possible combinations of the values of these variables.

A model can be considered consistent with a reference class if, when you average the model’s predictions over elements in that reference class, you are within some error epsilon of the true rate over that reference class.

One way to then think about predictive multiplicity is a case where we have at least two models that are consistent on all of the reference classes for some error bound epsilon, but that frequently make different predictions. In Roth and Tolbert’s characterization, this means that the probability that their predictions differ by more than some small amount epsilon is greater than epsilon. 

Now, returning to the previous question, should we accept such multiplicity as a fact of life in machine learning, an inevitable symptom of an underspecified learning problem? 

The recent work by Roth and Tolbert and a more technical version by Roth et al. argues that this kind of multiplicity is always resolvable, at least in theory. The answer to finding resolution lies in multicalibration, which I’ve discussed previously on the blog. A multicalibrated model is one that, for any efficiently identifiable, possibly intersecting set of groups that are supported by the data, its predictions are approximately calibrated. 

Multicalibration can be achieved by algorithms that work via a boosting process. For any reference class with sufficient probability mass on which the model is not found to be consistent, with access to data sampled from the same distribution, we can produce a new model with a lower squared error. We continue doing this until we arrive at a model that is consistent with all of the reference classes.

This leads Roth and colleagues to argue that: 

although individual probabilities are unknowable, they are contestable via a computationally and data efficient process that must lead to agreement. Thus we cannot find ourselves in a situation in which we have two equally accurate and unimprovable models that disagree substantially in their predictions—providing an answer to what is sometimes called the predictive or model multiplicity problem. 

In other words, they show that with sufficient data we can improve one or both of the models until we have a multicalibrated model. Put this way, this is not so surprising. The contributions of these papers are more nuanced of course, and including showing how bounds on the amount of data needed scales with parameters like the error bound and probability mass of the considered subsets. 

I like how framing predictive multiplicity in terms of models being inconsistent with reference groups nicely connects underspecification of individual probabilities with model multiplicity. This link has previously been left vague. My only real complaint (which applies to lots of theory papers related to calibration) is the downplaying of the distinction between theoretically possible and practically possible in some of the statements. E.g., in the quote above, saying “we cannot find ourselves in a situation in which we have two equally accurate and unimprovable models that disagree substantially in their predictions” requires some qualification, because we can absolutely find ourselves in a situation where we have multiple models that disagree and we don’t have sufficient data (because we’re dealing with rare classes/very large labels and limited data). 

One question this work has me now thinking about is when observing model multiplicity is still useful, even if you could reconcile the models to some extent through cross-calibration. It’s directly relevant to some work that Abhraneel Sarma, Dawei Xie and I have been doing related to Cynthia Rudin’s vision that predictive multiplicity is a good thing, e.g., because it provides room for identifying models that align with human preferences like fairness or monotonicity constraints on the relationship between features and outcome. Our part of this has been to develop an interactive interface (and more generally think through a workflow for incorporating domain expertise in model selection) for Rashomon sets of Generalized Additive Models. Here, the individual models in the set can differ in terms of what features they include, how they are coded, and what their shape functions look like. 

It seems that when a model’s predictions are intended as a decision aid for a human expert, then rather than trying to reconcile the multiplicity entirely by cross-calibrating, we may do better by deploying a model that aligns with the domain experts’ preferences. One reason that I’ve brought up before is that trusting the calibration data can seem to contradict the reasons for including humans in a decision process in the first place: often we want them there because we are wary of distributions shifting or the model predictions unknowingly reflecting artifacts in the training data. Involving the expert in model selection to ensure the deployed model captures key aspects of how human experts think features relate to the outcome (e.g., having asthma should not result in lower risk) may produce better decisions in practice by providing some robustness against shifts, even if we could have improved calibration according to  historical data. There are also problems that arise when experts have access to some highly performant model but it conflicts with their expectations about how features should be used. I’ve heard stories about this happening in medical settings where its discovered after a fancy new model is deployed that it’s basically being ignored.

Barnabas Szaszi writes:

I’m contacting you now to get your advice/interpretation on the piranha theorems regarding a paper I’m writing about the generalizability of behavioral interventions. Here, I use the Piranha theorem to back my claim that average affect of nudges are small:

First, there are theoretical reasons to assume that the average effect of choice architecture interventions is modest. It has been widely argued that in systems such as human behavior – and more broadly in social and behavioral sciences – phenomena are causally dense (Meehl, 1967, Gelman, 2011, Almaatouqa et al. 2022). The Piranha theorem (Gelman 2017 blogpost, Tosh et al. 2025) suggests that in such systems, many influencing factors operate concurrently, the interaction and interference of these factors would overwhelm most main effects leading to predominantly small effects.

I was wondering if you could help to clarify whether my interpretation is correct, these effects remain usually individually small because they overwhelm each other, or they are rather unlikely to remain individually small because of their cumulative impact?

My reply:

The basic piranha paradox is that there are subfields where many papers claim to find large and general effects, but of different factors. In the wild, where all these factors are operating in an uncontrolled fashion, the result of adding all these large effects, if these effects are independent, is that outcomes will be extremely variable. Given that real-world behavior is not so unpredictable, this implies that, either there are not a large number of large effects, or that the large effects happen to have large negative interactions that cause them to cancel out. But in that latter case the original claim (that effects are general) is false.

It’s also correct to say that many small effects won’t remain small, in the sense that, even when researchers talk about small effects, these effects are not so small. For example, an intervention that is claimed to increase some desirable behavior by 10%. That sounds small, but given that any intervention won’t work on most people, an average treatment effect of 0.1 is still large; see here. If you have many different treatments, each with an average effect of 0.1, then the piranha issue still arises: in the wild, with these treatments operating in an uncontrolled way, all these effects will add up in unexpected ways, leading to complete chaos–unless the effects interact in a way to mostly cancel each other out, in which case the effects of each treatment are highly context-dependent, in a way that would cast doubt on the original claims of universality.

This is Jessica. What’s the common assumption behind the following? 

• • Partial pooling of information over groups in hierarchical Bayesian models 
  • In causal inference of treatment effects, saying that the outcome you would get if you were treated (Y^a) shouldn’t change depending on whether you are assigned the treatment (A) or not
  • Acting as if we believe a probability is the “objective chance” of an event even if we prefer to see probability as an assignment of betting odds or degrees of belief to an event

The question is rhetorical, because the answer is in the post title. These are all examples where statistical exchangeability is important. Exchangeability says the joint distribution of a set of random variables is unaffected by the order in which they are observed. 

Exchangeability has broad implications. Lately I’ve been thinking about it as it comes up at the ML/stats intersection, where it’s critical to various methods: achieving coverage in conformal prediction, using counterfactuals in analyzing algorithmic fairness, identifying independent causal mechanisms in observational data, etc. 

This week it came up in the course I’m teaching on prediction for decision-making. A student asked whether exchangeability was of interest because often people aren’t comfortable assuming data is IID. I could see how this might seem like the case given how application-oriented papers (like on conformal prediction) sometimes talk about the exchangeabilty requirement as an advantage over the usual assumption of IID data. But this misses the deeper significance, which is that it provides a kind of practical consensus between different statistical philosophies. This consensus, and the ways in which it’s ultimately limited, is the topic of this post.

Interpreting the probability of an individual event

One of the papers I’d assigned was Dawid’s “On Individual Risk,” which, as you might expect, talks about what it means to assign probability to a single event. Dawid distinguishes “groupist” interpretations of probability that depend on identifying some set of events, like the frequentist definition of probability as the limiting frequency over hypothetical replications of the event, from individualist interpretations, like a “personal probability” reflecting the beliefs of some expert about some specific event conditioned on some prior experience. For the purposes of this discussion, we can put Bayesians (subjective, objective, and pragmatic, as Bob describes them here) in the latter personalist-individualist category. 

On the surface, the frequentist treatment of probability as an “objective” quantity appears incompatible with the individualist notion of probability as a descriptor of a particular event from the perspective of the particular observer (or expert) ascribing beliefs. If you have a frequentist and a personalist thinking about the next toss of a coin, for example, you would expect the probability the personalist assigns to depend on their joint distribution over possible sequences of outcomes, while the frequentist would be content to know the limiting probability. But de Finetti’s theorem shows that if one believes a sequence of events to be exchangeable, then you can’t distinguish their beliefs about those random variables from conceiving of independent events with some underlying probability. Given a sequence of exchangeable Bernoulli random variables X1, X2, X3, … , you can think of a draw from their joint distribution as sampling p ~ mu, then drawing X1, X2, X3, … from Bernoulli(p) (where mu is a distribution on [0,1]). So the frequentist and personalist can both agree, under exchangeability, that p is meaningful for decision making. David Spiegalhalter recently published an essay on interpreting probability that he ended by commenting on how remarkable this pragmatic consensus is.

But Dawid’s goal is to point out ways in which the apparent alignment is not as satisfactory as it may seem in resolving the philosophical chasm. It’s more like we’ve thrown a (somewhat flimsy) plank over it. Exchangeability may sometimes get us across by allowing the frequentist and personalist to coordinate in terms of actions, but we have to be careful how much weight we put on this.  

The reference set depends on the state of information

One complication is that the personalist’s willingness to assume exchangeability depends on the information they have. Dawid uses the example of trying to predict the exam score of some particular student. If they have no additional information to distinguish the target student from the rest, the personalist might be content to be given an overall limiting relative frequency p of failure across a set of students. But as soon as they learn something that makes the individual student unique, p is no longer the appropriate reference for the individual student’s probability of passing the exam. 

As an aside, this doesn’t mean that exchangeability is only useful if we think of members of some exchangeable set as identical. There may still be practical benefits of learning from the other students in the context of a statistical model, for example. See, e.g., Andrew’s previous post on exchangeability as an assumption in hierarchical models, where he points out that assuming exchangeability doesn’t necessarily mean that you believe everything is indistinguishable, and if you have additional information distinguishing groups, you can incorporate that in your model as group-level predictors.

But for the purposes of personalists and frequentists agreeing on a reference for the probability of a specific event, the dependence on information is not ideal. Can we avoid this by making the reference set more specific? What if we’re trying to predict a particular student’s score on a particular exam in a world where that particular student is allowed to attempt the same exam as many times as they’d like? Now that the reference group refers to the particular student and particular exam, would the personalist be content to accept the limiting frequency as the probability of passing the next attempt? 

The answer is, not necessarily. This imaginary world still can’t get us to the generality we’d need for exchangeability to truly reconcile a personalist and frequentist assessment of the probability. 

Example where the limiting frequency is constructed over time

Dawid illustrates this by introducing a complicating (but not at all unrealistic) assumption: that the student’s performance on their next try on the exam will be affected by their performance on the previous tries. Now we have a situation where the limiting frequency of passing on repeated attempts is constructed over time. 

As an analogy, consider drawing balls from an urn, where when we draw our first ball, there is 1 red ball and 1 green ball in it. Upon drawing a ball, we immediately return and add an additional ball of the same color. At each draw, each ball in the urn is equally likely of being drawn, and  the sequence of colors is exchangeable. 

Given that p is not known, which do you think the personalist would prefer to consider as the probability of a red ball on the first draw: the proportion of red balls currently in the urn, or the limiting frequency of drawing a red ball over the entire sequence? 

Turns out in this example, the distinction doesn’t actually matter: the personalist should just bet 0.5. So why is there still a problem in reconciling the personalist assessment with the limiting frequency?

The answer is that we now have a situation where knowledge of the dynamic aspect of the process makes it seem contradictory for the personalist to trust the limiting frequency. If they know it’s constructed over time, then on what ground is the personalist supposed to assume the limiting frequency is the right reference for the probability on the first draw? This gets at the awkwardness of using behavior in the limit to think about individual predictions we might make.

Why this matters in the context of algorithmic decision-making

This example is related to some of my prior posts on why calibration does not satisfy everyone as a means of ensuring good decisions. The broader point in the context of the course I’m teaching is that when we’re making risk predictions (and subsequent decisions) about people, such as in deciding who to grant a loan or whether to provide some medical treatment, there is inherent ambiguity in the target quantity. Often there are expectations that the decision-maker will do their best to consider the information about that particular person and make the best decision they can. What becomes important is not so much that we can guarantee our predictions behave well as a group (e.g., calibration) but that we understand how we’re limited by the information we have and what assumptions we’re making about the reference group in an individual case.