This is Jessica. I got through a long string of deadlines and invited talks and now I’m back to thinking about calibration and decision-making.
In a previous post I was wondering about the relationship between calibration and Bayesian use of information – Bayesian theory implies calibration, but when does calibration imply Bayesian use of information? I am still interested in this question, and may follow up with a post on this topic later.
But more broadly I have been considering what to do with some recent observations in theoretical computer science involving calibration and decision-making. For example, what’s the significance of observations on the sufficiency (or lack thereof, depending on how you look at it) of calibration for good decisions? Or the possibility of identifying predictors that we expect to be calibrated for many downstream decision tasks, or over many possibly intersecting groups represented in the data? How realistic is it to expect to achieve these solutions in practice?
I think these questions are important in light of renewed interest in calibration. Rather than stuff all of this in a single blog post, today I’ll discuss a few issues with embracing calibration as the goal when predictions are inputs to human experts’ decisions.
Calibration is enough for good decisions in a restricted sense
Consider the following:
“Calibrated predictions have the property that it is optimal for a decision maker to optimize assuming that the prediction is correct.”
“Calibration promises that, simultaneously for all downstream problems, the policy that treats the predictions of a calibrated model as correct and acts accordingly is uniformly best amongst all policies.”
It would be easy to conclude from these statements that “Calibration is all you ever need for good decisions.” But the above statements assume a decision-maker who has no access to additional contextual information other than the prediction. In such a case, if you know the probability is calibrated, then the decision-maker can’t do better than assuming it is the true probability in making their decisions.
Stated this way, the sufficiency of calibration for decision-making can seem to border on tautological: If you evaluate decisions using expected value and assume the decision-maker will only decide based on the prediction from your model, then calibration is enough to trust the predictions. The result is partly a function of how we are evaluating things.
If, on the other hand, you think a decision-maker may have access to additional information about a decision instance (that is not available to the model making the predictions), then predictions that are calibrated with respect to the data that the model can access but not this additional information are not necessarily enough. See, e.g., this demonstration by Corvelo Benz and Gomez Rodriguez of how predictions calibrated over the model’s information are not always sufficient for a decision-maker with access to additional features to discover the optimal policy.
It’s interesting how casual statements that “calibration is all we need for good decisions” can also seem to contradict aspirations for integrating predictive models in many cases where human experts currently make decisions. In practice, it seems that there’s often an expectation that a human expert might have additional information over a statistical model. And so we don’t just want the human expert to blindly trust model predictions. If that were the case, we could just automate things.
Part of this relates to limitations on how we know that a model is calibrated in practice. I expect it’s still fairly common when seeking to deploy AI models to help decision-makers in fields like medicine to evaluate model calibration prior to the human expert using it in practice, and maybe sporadically at most after that. For example, we might do checks on calibration on data from the training/test distribution. But often we maintain some skepticism about whether we got the problem definition right. So part of the assurance provided by having a human on hand is that they might be able to recognize when the model’s assumptions no longer hold, even when the model “thinks” its perfectly calibrated.
There’s also often an expectation that a doctor, for example–through interacting with a patient to gain more details, or observing hard to formalize signals like their mood–has the potential to make a better decision than the model could have made alone by combining their own information with that contained in model predictions.** If we allow for the possibility that an expert consulting model predictions might have access to other relevant information, the value of calibration gets more complicated than suggested by the statements above.
Calibration says nothing about how good individual decisions are.
Another human-facing issue that doesn’t often get acknowledged in the theoretical literature on calibration is that often decision-makers must care about being accountable or legally liable for individual decisions. For example, doctors, and the organizations that employ them, need to be able to demonstrate a lack of negligence for specific decisions if pressed. This might entail showing that given the available information, one could not have made a better or less harmful decision.
One way to define perfect calibration uses “outcome distinguishability.” This says that the true distribution of individual-outcome pairs (x, y*) is indistinguishable from the distribution of pairs (x, ytilde) generated by simulating a y using the true (empirical) probability, ytilde ∼ Ber(ptilde(x)). This helps illustrate that calibration is subject to multiplicity: there may be multiple predictors that are outcome indistinguishable from the true distribution but which assign different predictions to individual outcomes. We can achieve calibration while still producing predictions that are biased, for example. So knowing predictions are calibrated doesn’t tell us about the limitations of the available information for making a decision on a particular instance. What would be better is something Berk Ustun and I have been thinking a lot about recently.
That calibration isn’t sufficient for good individual decisions is also apparent if you consider the existence of algorithms that achieve calibration in adversarial, online (i.e., sequential) settings, where calibration can be achieved, for example, by injecting noise into predictions. There’s a difference between making good individual decisions and achieving calibration via clever posthoc error accounting.
I would hope these points are clear enough from reading the emerging work on calibration. But I’m not sure if they always are. Overall, I’m excited that calibration for decision-making is a popular topic, and I think there’s a lot to still characterize. I’m just not sure we’ve developed the processes or theory to know how valuable these results are in practice, particularly in cases where predictions inform human experts. So there’s a risk that framing calibration as sufficient for good decisions fixates attention on the wrong thing.
**Whether this folk belief is justified by evidence or philosophy is a separate question worth discussing in its own post.
It would be really useful to define “calibration” here!
Oh, sure! From my previous post:
Calibration is typically understood as the property of a prediction model or algorithm where for all predictions that are approximately b, the event realizes at a rate of b, and this is true for all values of b (where b is binned to some extent). It can be defined over a distribution or over a sequence.
This is all so embedded in a Frequentist assumption and background.
What is the rate for “there’s a 30% chance of rain in Los Angeles on Nov 1 2024?”
If we instead think like a Bayesian, and we realize we are maybe in some instances interested in *rates* then we think about creating a model which predicts a *rate* then what you want is that the probability of rates near the true rate is 1.
Suppose for example we’re interested in predicting the rate of daily rainfall over 1 inch in the next 1000 days.
Suppose our model says there’s a 99% chance this rate is between 1 and 3 per 1000 days. Suppose we observe a 1000 days of record and we find it’s 3… Then we’ve got a sharp prediction that is successful.
Suppose instead we say there’s a 99% chance this rate is between 2 and 400… Then the prediction is successful but not sharp.
Yes.
It’s interesting to think about what the Bayesian does when there is no stable process (e.g., in the adversarial, online case). I think the answer is probably nothing? Meanwhile the frequentist designs and studies algorithms (including determistic ones: https://arxiv.org/abs/2210.07152) for achieving calibration. I’ve never been quite sure what to make of this kind of work. For instance, are there real world motivating applications where this is what we really need?
I think @Daniel Lakeland and @Jessica Hullman are saying the same thing. And that’s the way it’s usually formulated. It is frequentist, or what I call “pragmatic Bayes” (to distinguish it from “subjective Bayes” and “objective Bayes”). The nice thing about Bayes is that it’s calibrated in expectation when you use the true data generating model. That’s why we can do simulation-based calibration checking on our inferences.
I like the presentation in the following, which goes through several notions of calibration, all of which match the intuitive definitions from Jessica and Daniel above, but differ in subtle ways.
Gneiting et al. 2007. Probabilistic forecasts, calibration, and sharpness. JRSS-B. https://academic.oup.com/jrsssb/article/69/2/243/7109375.
Just as importantly, they talk about sharpness, which is what Daniel’s getting at. Sharp probabilistic predictions are near 0 or 1 for a fixed event (Daniel’s changing the events and keeping the probabilities the same).
Calibration over a sequence of events, as @chris mentions below, can be very poor measures of what’s going on in subgroups of those events. For instance, you can be calibrated predicting yearly weather using yearly statistics, but if you restrict the set of predictions to just December, those year-calibrated predictions won’t be month-calibrated.
Jessica—I missed your response. There doesn’t need to be a stationary process to measure calibration. All we need to measure calibraiton is predictions of probabilities for each of a sequence of events and then indicators of whether or not they occurred.
Calibration was already being studied in the 1980s. Phil Dawid’s paper “The Well-Calibrated Bayesian” is from 1982 (paywalled by JASA, but a quick search turns up pirated versions). It’s an interesting precursor to workflow in that it discusses iterative model refinement as you reject calibration tests.
Hi Bob,
Yes, I know there doesn’t need to be a stationary process. That’s what I’m pointing out – we can get calibration over a sequence even when we assume Nature is picking the outcomes adversarially (ie to maximally mess with our predictions).
And I like the Well-calibrated Bayesian paper, I referred to it in my last post on calibration and decisions.
Bob, Jessica,
I want to just re-emphasize that probability measures credibility, frequency measures a rate, and individual outcomes are unique. Let’s take the simplest case I can think of that isn’t a binomial (in this case it’s gonna be a multinomial secretly).
We point a detector at a galaxy far away. We have a spectrometer which measures the energy of individual photons it receives. This works by diffracting the photons and so spreading them out along a line, and then having CCD type detectors that acquire charge when a photon hits them. (I’m making it up, but basically it’s along the right lines, there’s a linear stripe of detectors and the photon only hits one of the detectors and that tells us the wavelength).
We have a model which describes the rate at which photons will be detected for each bin of wavelength. Let’s say there’s 1000 bins between 100nm and 10nm or something (these would be UV to far UV I guess). This rate is parameterized as a curve f(lambda).
Now our model has say 6 parameters which describe the processes going on in the galaxy, and for each vector of 6 numbers q we get a curve f(lambda), and by discretizing the ranges of lambda from the curve we can calculate a predicted rate for each bin of the spectrometer f_i with i going from 1 to 1000 and each i representing a different range of wavelengths lambda.
But, of course, we don’t know the values of the q parameters which correctly describe the galactic processes, so we have a prior over q and we therefore have a prior-predictive distribution over the frequencies f_i. That is, a **probability distribution** over the *frequencies* f_i.
Now we collect 200 photons and so we have counts of observed f_i, from this data we get posterior distributions over the q (the posterior prefers q values that lead to f_i values that are near the observed ones) so we have a **posterior** predictive distribution over f_i, the frequency with which we expect **future** photons to land in the detector in each of the bins.
Suppose we draw a sample of 100 vectors of q from the posterior, and we use this to approximate the full distribution. For each q, there is a curve f(lambda) and from that an induced f_i.
Graphically, we can represent the f_i as a little “skyline plot” (a histogram). Since we have 100 samples from q, we also have 100 skylines. We could spaghetti plot these skylines. Now graphically we can see the *probability* that the *frequency* of photons landing in bin i = 30 is some range of values…
Like, “there is a 90% chance that the fraction of photons landing in bin 30 will be between 40 and 50% of the total photons captured”
There is *zero* sense in which “90%” is a “calibrated” probability. It’s not even in principal observable. It describes “how much credence we give on the basis of our model and data that the future observed values would be near 45% in this bin”. As Jaynes says, asking about calibration here is like doing measurements of a dog to find out how much a boy loves it.
There is absolutely a sense in which 40-50% is a calibrated *frequency* of the future observations. In the sense that if we observe a bunch more photons and find that only 7% were in this bin, or as much as 94% were in the bin or something, we’d say that the model failed.
But note, we don’t have a single point estimate of the frequency f_30, we have a *distribution* over the frequency f_30.
Thanks Daniel, I find the example very useful to separate empirical frequency and degrees of belief.
On not having a single point estimate… I think you’re just pointing out that the Bayesian gets the distribution while the frequentist might only get the point estimate. Or are you also implying that a Bayesian would never be comfortable summarizing their predicted distribution with a point? I assume whether they would will depend on the downstream decision problem they face.
A remaining question that’s been at the back of my mind is philsophical. If the Bayesian knew that the next observed f_i would always be chosen to be maximally surprising/minimum likelihood according to their predicted distribution, would they go through the trouble of constructing a model in the first place? It seems they would be starting with an uninformative prior and wouldn’t expect to learn anything that would reduce their uncertainty. Would they still perceive the Bayesian probability they can put over the frequencies as having some value in the adversarial setting? Maybe this question also depends on what sort of downstream decision problems they might face.
>On not having a single point estimate… I think you’re just pointing out that the Bayesian gets the distribution while the frequentist might only get the point estimate.
More than that, I’m pointing out that to the Frequentist, the “probability is the frequency”.
To the Bayesian, we have a probability distribution *over* the frequency. The Frequency is the thing we’re estimating and is not a probability, the probability is the thing we’re using to quantify how much uncertainty we have about the frequency.
If we have very little uncertainty about the frequency after observing bajillions of photons, then of course we can use a point estimate. But this is the asymptotic regime for large data. Conceptually, there’s still a difference between the probability and the frequency. In the asymptotic regime, there’s probability 1 that the frequency of f_30 is 45.000% for example.
Got it, thanks.
This discussion of calibration is also important for issues of equity. The analysis of the COMPAS recidivism model showed that well calibrated models can have very different impact on demographic groups when measured by other metrics like false positive rates at a specific threshold.
In my organization, there is a “calibration is all you need” group versus “calibrated models may be unfair”. I think your post helps me (in the latter group) in conversations with the former group. As part of of an AI compliance team, I couldn’t agree more on the link to the BU law article on negligence. I call this ‘its not the math that matters’.
The mathematical evaluations of a model are helpful, necessary components to a robust ethical review but we are far removed from them being sufficient.
Yes, what’s interesting is that multicalibration has become a popular notion of calibration motivated by fairness concerns. But whether it’s realistic in many settings where we’re making predictions about people I’m not sure.
https://proceedings.mlr.press/v80/hebert-johnson18a/hebert-johnson18a.pdf
Love this, looking forward to the other posts on the topic.
In a slightly broader view, I think it is often unappreciated that to show that your model actually lets people make better decisions you need to do a trial randomizing decisions/decision makers to “model” vs “standard” conditions and compare the outcomes of the decisions. Or at least approximate such trial.
Even models with very good predictions can be completely useless, because e.g. no such prediction is actually needed at the point the model is able to make it or the prediction is not actionable or the model does not fit well into the existing workflow of the decision maker. In the medical context it turns out that those are very difficult hurdles to overcome. Some more discussion of this can be found e.g. in van Royen et al. 2022: “Developing, validating, updating and judging the impact of prognostic models for respiratory diseases”.
One example I like is a sepsis warning system that did in fact reduce time to antibiotics administration from 3 hours to 2.3 hours (median time) with a very slight mortality benefit – this is not much, but was still very hard to achieve, even with a decently calibrated and working model (Tarabichi et al. (2021), Improving Timeliness of Antibiotic Administration Using a Provider and Pharmacist Facing Sepsis Early Warning System in the Emergency Department Setting)
Links to papers (separately, to avoid the parent being marked as spam):
van Royen et al. 2022 (validating models): https://doi.org/10.1183/13993003.00250-2022
Tarabichi et al. 2021 (a succesful randomized trial of a good model): https://doi.org/10.1097/CCM.0000000000005267
Thanks for these references, looking forward to reading them all.
I agree on the need for rigorous evaluation. Reminds me of this recent paper I saw presented, applying causal inference to evaluate AI deployments, e.g., to support judges’ decisions: https://arxiv.org/pdf/2403.12108
We have some related work where we apply statistical decision theory to identify the best attainable performance of a human and a predictive model, as well as baseline performance (the better of the two agents alone). We can then look at where observed performance of the human with the model falls: https://arxiv.org/abs/2401.15356
But this doesn’t really get around the need for randomization in cases where some outcomes might be censored (e.g., recidivism decisions, where we can’t observe what would have happened if a judge decides to convict a defendant).
Quick disclaimer: I come to this topic from the perspective of probabilistic forecasting and am sometimes a bit unsure how the understanding of calibration in there relates to how it is understood in other fields, especially computer science.
Usually in forecasting when we say calibration I think most people understand it as probabilistic calibration in the sense “all predictions that are approximately b, the event realizes at a rate of b, and this is true for all values of b”. This is however only one notion and for some applications (especially in climate and weather science), sometimes things like marginal calibration (equivalence of average forecast cdf and marginal/unconditional distribution) can be of similar or even greater importance. For example, if I want to use weather forecast model runs to probe possible and realistic extreme events I need marginal, but not probabilistic calibration (and they don’t imply eachother unfortunately). Both marginal and probabilistic calibration are actually implied by autocalibration (I usually understand it as calibration conditional on arbitrary information sets), which is a strictly stronger condition (see Gneiting and Resin, 2023), but which we usually cannot really verify – so most people in the forecasting literature stick with the notions above or intermediate notions.
So, when saying that calibration matters for decisions in my opinion this really depends on which notion of calibration. In addition, some authors (see e.g. Allen et al., 2024) have pointed out that we might be probabilistically calibrated overall, but not in the tail of the distribution, so when conditioning on high impact events. So, in some sense the forecast might be adequate for decisions, but not decisions for the tail of the outcome distribution. This can’t happen under autocorrelation but can with the weaker notions of calibration we can verify in practice. Another example is Bouttier and Marchal, 2024 who discuss data-driven derivation of probability thresholds to issue weather warnings. For this calibration is usually not needed (or even desirable), as we want some degree of overforecasting of high impact weather events (the effects of a false negative are much worse than the ones of a false positive). We can of course get this overforecasting also from calibrated forecasts, but calibration is not needed, nor might it be sufficient as we only need calibration to a certain degree whilst sharpness is actually more important in practice I would argue.
All of this said, if different notions of calibration are important for different applications, I still usually find it hard to figure out which notion of calibration is important for which application. In my own work I have been finding surprising non-equivalences where a model would seem calibrated overall, but then specifically mis-calibrated for things that I would actually be interested in. That and the lack of practical diagnostics to verify some notions of calibration can make it hard to do effective model checks sometimes.
Some references to the above:
Allen, S. et al. (2024) ‘Tail calibration of probabilistic forecasts’. Available at: https://doi.org/10.48550/arXiv.2407.03167.
Bouttier, F. and Marchal, H. (2024) ‘Probabilistic short-range forecasts of high-precipitation events: optimal decision thresholds and predictability limits’, Natural Hazards and Earth System Sciences, 24(8), pp. 2793–2816. Available at: https://doi.org/10.5194/nhess-24-2793-2024.
Gneiting, T. and Resin, J. (2023) ‘Regression diagnostics meets forecast evaluation: conditional calibration, reliability diagrams, and coefficient of determination’, Electronic Journal of Statistics, 17(2), pp. 3226–3286. Available at: https://doi.org/10.1214/23-EJS2180.
Yes – there are different definitions of calibration and which one we are assuming will definitely influence what conclusions we can draw about decisions from it. Usually when we say calibrated predictions in CS we mean approximately calibrated assuming some binning.
Interesting weather examples, thanks for sharing the papers!
From the Gneiting and Resin paper, it seems like autocalibration is like what I’m mentioning above as an outcome indistguishability definition of perfect calibration.. “in the sense of auto-calibration, meaning that the outcomes are indistinguishable from random draws from the posited distributions.”
>if different notions of calibration are important for different applications, I still usually find it hard to figure out which notion of calibration is important for which application
Agree. This is why I think the topic is rich in terms of there being lots to still characterize.
Dear Jessica,
Two things:
1. If the ‘decision’ = what is the optimal sequence to work in with a fixed capacity, calibration doesn’t matter.
2. Calibration for most classifier models is over some bin that, depending on the error rate, has to be big enough to smooth things out. This means each prob. is still ‘wrong’, and I can see cases where people are wary of using the system because the individual prob. looks off. I also want to puzzle over a classifier built on noise where calibration = global average.
Let’s trade-off between two models — both are calibrated but one with worse forecast error than the other.
This is w.r.t. your point: “Something I don’t say in this post but which relates to calibration being unsatisfying from a Bayesian perspective is that calibration doesn’t say anything about how well we communicate uncertainty for a specific decision. Knowing a forecast is calibrated doesn’t give me information about how uncertain the forecaster is about any specific forecast they are giving me. For that I need to know more about how the forecasts are constructed / how they change with more information.”