This is Jessica. A couple weeks I wrote a post in response to Ben Recht’s critique of conformal prediction for quantifying uncertainty in a prediction. Compared to Ben, I am more open-minded about conformal prediction and associated generalizations like conformal risk control. Quantified uncertainty is inherently incomplete as an expression of the true limits of our knowledge, but I still often find value in trying to quantify it over stopping at a point estimate.
If expressions of uncertainty are generally wrong in some ways but still sometimes useful, then we should be interested in how people interact with different approaches to quantifying uncertainty. So I’m interested in seeing how people use conformal prediction sets relative to alternatives. This isn’t to say that I think conformal approaches can’t be useful without being human-facing (which is the direction of some recent work on conformal decision theory). I just don’t think I would have spent the last ten years thinking about how people interact and make decisions with data and models if I didn’t believe that they need to be involved in many decision processes.
So now I want to discuss what we know from the handful of controlled studies that have looked at human use of prediction sets, starting with the one I’m most familiar with since it’s from my lab.
In Evaluating the Utility of Conformal Prediction Sets for AI-Advised Image Labeling, we study people making decisions with the assistance of a predictive model. Specifically, they label images with access to predictions from a pre-trained computer vision model. In keeping with the theme that real world conditions may deviate from expectations, we consider two scenarios: one where the model makes highly accurate predictions because the new images are from the same distribution as those that the model is trained on, and one where the new images are out of distribution.
We compared their accuracy and the distance between their responses and the true label (in the Wordnet hierarchy, which conveniently maps to ImageNet) across four display conditions. One was no assistance at all, so we could benchmark unaided human accuracy against model accuracy for our setting. People were generally worse than the model in this setting, though the human with AI assistance was able to do better than the model alone in a few cases.
The other three displays were variations on model assistance, including the model’s top prediction with the softmax probability, the top 10 model predictions with softmax probabilities, and a prediction set generated using split conformal prediction with 95% coverage.
We calibrated the prediction sets we presented offline, not dynamically. Because the human is making decisions conditional on the model predictions, we should expect the distribution to change. But often we aren’t going to be able to calibrate adaptively because we don’t immediately observe the ground truth. And even if we do, at any particular point in time we could still be said to hover on the boundary of having useful prior information and steering things off course. So when we introduce a new uncertainty quantification to any human decision setting, we should be concerned with how it works when the setting is as expected and when it’s not, i.e., the guarantees may be misleading.
Our study partially gets at this. Ideally we would have tested some cases where the stated coverage guarantee for the prediction sets was false. But for the out-of-distribution images we generated, we would have had to do a lot of cherry-picking of stimuli to break the conformal coverage guarantee as much as the top-1 coverage broke. The coverage degraded a little but stayed pretty high over the entire set of out-of-distribution instances for the types of perturbations we focused on (>80%, compared to 70% for top 1- and 43% for top 1). For the set of stimuli we actually tested, the coverage for all three was a bit higher, with top 1 coverage getting the biggest bump (70% compared to 83% top 10, 95% conformal). Below are some examples of the images people were classifying (where easy and hard is based on the cross-entropy loss given the model’s predicted probabilities, and smaller and larger refers to the size of the prediction sets).
We find that prediction sets don’t offer much value over top-1 or top-10 displays when the test instances are iid, and they can reduce accuracy on average for some types of instances. However, when the test instances are out of distribution, accuracy is slightly higher with access to prediction sets than with either top-k. This was the case even though the prediction sets for the OOD instances get very large (the average set size for “easy” OOD instances, as defined by the distribution of softmax values, was ~17, for “hard” OOD instances it was ~61, with people sometimes seeing sets with over 100 items). For the in-distribution cases, average set size was about 11 for the easy instances, and 30 for the hard ones.
Based on the differences in coverage across the conditions we studied, our results are more likely to be informative for situations where conformal prediction is used because we think it’s going to degrade more gracefully under unexpected shifts. I’m not sure it’s reasonable to assume we’d have a good hunch about that in practice though.
In designing this experiment in discussion with my co-authors, and thinking more about the value of conformal prediction to model-assisted human decisions, I’ve been thinking about when a “bad” (in the sense of coming with a misleading guarantee) interval might still be better than no uncertainty quantification. I was recently reading Paul Meehl’s clinical vs statistical prediction, where he contrasts clinical judgments doctors make based on intuitive reasoning to statistical judgments informed by randomized controlled experiments. He references a distinction between the “context of justification” for some internal sense of probability that leads to a decision like a diagnosis, and the “context of verification” where we collect the data we need to verify the quality of a prediction.
The clinician may be led, as in the present instance, to a guess which turns out to be correct because his brain is capable of that special “noticing the unusual” and “isolating the pattern” which is at present not characteristic of the traditional statistical techniques. Once he has been so led to a formulable sort of guess, we can check up on him actuarially.
Thinking about the ways prediction intervals can affect decisions makes me think that whenever we’re dealing with humans, there’s potentially going to be a difference between what an uncertainty expression says and can guarantee and the value of that expression for the decision-maker. Quantifications with bad guarantees can still be useful if they change the context of discovery in ways that promote broader thinking or taking the idea of uncertainty seriously. This is what I meant when in my last post I said “the meaning of an uncertainty quantification depends on its use.” But precisely articulating how they do this is hard. It’s much easier to identify ways calibration can break.
There a few other studies that look at human use of conformal prediction sets, but to avoid making this post even longer, I’ll summarize them in an upcoming post.
P.S. There have been a few other interesting posts on uncertainty quantification in the CS blogosphere recently, including David Stutz’s response to Ben’s remarks about conformal prediction, and on designing uncertainty quantification for decision making from Aaron Roth.
Jessica, do you have any TL;DR versions of your posts? It might just be my poor attention span or lack of expertise in your field but I’m often intrigued by the first few sentences of your posts and then quickly lose the thread. That suggests you might be able to engage more people in your work with some more concise writing. Thanks for considering!
Perhaps the problem is that I don’t blog to argue a point, I blog to think through a topic or question.
Too much time has been wasted on dense jargon about “guarantees” based on silly assumptions.
I am 100% confident that whatever is being guaranteed here is something no one cares about.
That may be. Though so far my experience with conformal prediction is that the first time we applied in my lab (this image labeling scenario) the coverage of the conformal sets held up better to all the perturbations we tried. So while I agree its overhyped, I’m open to the possibility that there may be some real world applications where it has some value for expressing uncertainty to human decision makers.
If you can perform an impressive feat or predict something surprising about the future with it I will reconsider.
Your experimental setup seems to envision that predictive models will be used by non-experts, that the algorithms are “black boxes” and users can only access evidence of how well they predict, not any information about how these boxes actually work. I do believe that is the way many such predictive models are being used and may continue to be used – for example, doctors may have access to the output of these models to aid their decision making. I agree that it is worthwhile to investigate when and how (and how much) people would rely on model outputs, including measures of uncertainty, when making decisions.
But what about decision makers who understand something about how the models work? The predominant paradigm seems to be that only algorithm outputs are available to decision makers. Is that the best way to envision using algorithms to complement human decision-making? Is there a role for training decision makers in understanding the modeling methodologies? I ask this because I mostly teach graduate business students who are not likely to build these models themselves, but will be in the position of using (or not using) the outputs of these models. My approach is that decision makers should have some experience with the inputs to these models and how the models work – at a high intuitive level – if they are going to productively use the outputs of these models. These experiments seem to suggest a different reality – where decision makers don’t understand much about how the models work, but can only access evidence on their past output when they decide how and when to use model predictions. My question is what is the best way to envision potential complementarity between human and algorithmic decision making?
>The predominant paradigm seems to be that only algorithm outputs are available to decision makers. Is that the best way to envision using algorithms to complement human decision-making?
No, I don’t think it is. If this was a higher stakes decision than labeling images to train an ML, then presumably you’d want to give the end-users context on how the model was trained and some form of explanation (e.g., saliency map) to convey what it appears to be reacting to in the image. The image labeling task we studied is a low stakes easy-to-study setting. It’s a little contrived, though providing model assistance when people are labeling images for ML purposes is something that is done in practice sometimes so not entirely artificial. I’d rather we had studied some sort of medical decision making for instance, but the lead author needed to graduate and it’s hard to get access to physicians.
A better direction entirely for assisting human decisions with AI may be to make the prediction about what features of the instance (image, criminal defendant, etc) that are most likely to be discriminative, so the human can focus more attention on those. That would probably scale better to high stakes decisions.
This is a bit harsh for me IMO, but I haven’t had a real great use case for conformal predictions sets in my work instead of doing models for individual categories and basing subsequent decisions off of those individual probabilities.
For a simplified example, imagine you have a model predict {Cat,Dog,Human}, and this model says {0.8,0.19,0.01} for an image, so a 99% conformal set is {Cat,Dog}. The set predictions are always summed to 1.
Now imagine an alternative, I fit three individual models, so you get an independent probability for Cat or Dog or Human, and say those were {0.08,0.019,0.001}.
The second model to me has always been more useful in work applications than the “needs to sum to 1” multinomial model. I grant that I may have different work conditions than the examples Jessica shows here — can folks give other conformal examples than classifying images? For a few different examples where doing the individual models makes more sense IMO:
– Say you have an internet chatbot give diagnoses based on user symptoms. Someone may have a higher probability that the diagnosis is heart-burn than heart-attack, but the best course of action may be saying “you might be having a heart attack, please go to the hospital”.
– This application forecast risk for probationers: {violent,non-violent,no-crime}, https://projecteuclid.org/journals/annals-of-applied-statistics/volume-17/issue-1/Nested-conformal-prediction-sets-for-classification-with-applications-to-probation/10.1214/22-AOAS1650.short. It isn’t clear to me what the benefit of this is over having different models for violent and non-violent, and then basing subsequent decisions on those in a cumulative approach. Especially since these are not competing hazards.
Say for example you have someones probabilities:
violent: 0.1
non-violent: 0.8
And another
violent: 0.2
non-violent: 0.05
Say you think violent is 5 times worse than non-violent (or use whatever cost estimate you want). Here person 1 would have a total risk of 5*0.1 + 0.8 = 1.3, and the second person would have a total risk of 5*0.2 + 0.05 = 1.05.
So in this scenario, person 1 has a higher total risk. But change the multiplier to 10, and person 2 has a much higher risk. That to me is a better way to formulate this problem.
If folks have other use cases of conformal prediction sets I would be interested to hear them. The image classification just seems so cheeky I have a hard time translating it to other use cases.
Whoops – a bit harsh was meant in response to Anoneuoid’s comment above.
I had never heard the term “conformal prediction” until now. As soon as I tried figuring it out started to get flash backs to first learning NHST.
It has all the same red flags. Like why is this word “conformal” here in front of prediction (why is it a p-value instead of a probability)?”
Once I saw “guarantee” I knew its a bunch of pedantic deductions/calculations from silly premises that people will only bother with if they commit at least one logical fallacy.
Then I clicked one of the links and saw someone else essentially saying what my heuristics had gathered after 5 min, which got me to 100% confidence.
Nothing wrong with being more or less uncertain about “predictions”, but youre better off starting out with interpolation vs extrapolation than whatever this conformal stuff is.
My intuition is similar to yours. We have Cox’s theorem to tell us that if you’re generating intervals that are compatible with logical inference in the Cox sense, you’re doing Bayesian statistics. I’m happy with the Cox sense, and Conformal prediction isn’t Bayesian statistics, and so therefore it’s got some undesirable property ipso-facto. I’m not quite sure which undesirable properties it has, but it must have them, there’s a mathematical proof.
@Daniel: Alternatively you may not commit yourself to buying all of Cox’s postulates all the time. There’s no mathematical proof that says all that isn’t Bayes must be undesirable in some sense. Indeed mathematics doesn’t say anything about what’s desirable, it only deducts from certain premises, which are by no means uncontroversial in this case. (This is a general point; in the specific case of conformal prediction I think you will find something that you really don’t like if you look at it for long enough. ;-)
Christian, I’m curious what you have in mind when you say “I think you will find something that you really don’t like if you look at it for long enough” ….
Christian, oh sure I agree, not everyone has to be committed to Cox’s axioms to the extent that I am. And even I might not be committed to them all the time, perhaps I might at some point like to work with some logical axioms as a foundation that are different from propositional logic under some circumstances, like some kind of 3 valued logic or something. Generalizing 3 valued logic probably goes differently from Cox.
So far however, when it comes to science in the real world, I’ve been happy enough, and I find that it’s not too hard to find stupid things that other systems do such as Mayo’s Severity testing ( https://itschancy.wordpress.com/2019/02/05/the-sev-function-just-plain-doesnt-work/comment-page-1/ )
Jessica, he’s probably just saying that *I* am ornery and could find a problem with conformal prediction because I’m pretty strongly committed to Cox’s Theorem as being very useful, not that everyone would find some problem.
The problem I’m aware of when it comes to Conformal Prediction is that it claims only to satisfy a certain frequency of errors by outputting sets. As long as you’re making many repeated predictions all of which have very similar cost/benefit this might work out ok. But as soon as one particular problem is really really important, it falls apart. The Conformal Prediction gives you a set, but not a “measure” of which elements of the set are more or less credible. Credibility is specifically the point of Bayesian Probability under Cox’s axioms.
@Jessica: Well, if I understand conformal prediction well, the “conformal probabilities” are performance characteristics like in confidence intervals, and don’t allow to say that “the probability that the predicted value is in the set is so-and-so” *given the set*. And I guess that Daniel doesn’t like that. I personally don’t have that big a problem with it.
Thank you both for the context, Daniel and Christian. I’m still thinking through the various critiques I’m hearing so useful to hear the reactions.
@Daniel: From my point of view the issue in connection with statistical inference is not so much to question specific postulates of Cox (that can be done but doesn’t seem to be relevant here), but rather the question in what situation I am interested in a plausibility assignment according to Cox and in what situations I’m not. Cox’s theorem is about epistemic probability, whereas frequentist probability is aleatory, i.e., modelling the underlying data generating process rather than a state of information. I think it is legitimate to say that in a certain situation I care about frequentist probability and not epistemic probability, and then see what probability statements I can make regarding for example the uncertainty of parameter estimation, or prediction. I’m then outside the domain of Cox’s theorem. You can say that you prefer epistemic probabilities and that I should use them, too, which is also a legitimate point of view, but this is a very different kind of discussion from saying “something must be wrong with conformal prediction because it doesn’t satisfy Cox’s theorem”.
Christian:
If you are starting from a large set of measurements and fitting a distributional shape to it, then verifying the distributional shape continues to be a good model of future measurements (stationary distribution) and then asking questions about how often you might see future events that behave in a certain way then “aleatoric probability” is probably applicable. It’s also probably applicable if you are using an RNG to sample from a finite population of unchanging objects (like a lot of pill bottles or bolts). In my experience this is a vanishingly small fraction of real world science. It’s a decent model of thermal noise in electronics and quality control processes and such… but it’s a terrible model for human medical treatments or economic activities or politics or designed experiments in chemistry or whatever.
@Daniel: I think that science regularly uses models that in some respects are very different from the reality they aim at modelling. I also think that in any specific situation Bayesian models rather regularly deviate from the epistemic situation they claim to model, regardless of whether Cox’s theorem implies that if there is any “correct” epistemic model it has to be a Bayesian one. Therefore I don’t think that it is a valid argument against aleatory probability in general that reality isn’t quite like aleatory probability models say. This applies to all models. I have written on my position in some detail here: https://arxiv.org/abs/2007.05748
I don’t understand your example. I think your classification is totally based on point estimates: violent/non-violent probabilities. Conformal prediction would yield 4 sets of predictions: violent, non-violent, both, neither, all given some alpha level of confidence. The level of confidence can be based on some weighting of the risks associated with errors. But the use case is that your approach appears to result in two prediction sets: violent and non-violent, whereas the conformal prediction will almost always provide some cases that cannot be classified with that level of confidence.
I don’t understand “The level of confidence can be based on some weighting of the risks associated with errors”.
But how about this then — what exactly do we want to have guarantees for? In the probationer case, you maybe want something like “for those classified as low risk, I want the rate of violent to be less than x% and I want the rate of non-violent to be less than y%”. That is control the false negative rate for each category to some established standard. (You could have other objectives, but that seems to me reasonable offhand.)
In the case of the symptoms and the heart attack example, you might want to have high recall — e.g. we identify 99/100 heart attack cases.
It isn’t clear to me how the conformal sets help with this — or what exactly we want with the conformal sets. Can folks give a not image label example where we want k-sets to have some coverage? Am I confused and the conformal sets can accomplish my above examples in a way that is different than doing individual models?
You set your level of confidence (unfortunately, that word again) appropriate for your decision problem. So, if you want to limit the potential for false negatives you can do that – you’ll have to live with the resulting indeterminacy of classifications. That is, the more confident you want to be about the classifications, the less unambiguous cases will result. So you may end up with a lot of parole decisions that the model cannot help you with.
While the output of conformal predictions are sets, as Daniel says, there is also information about how close or far cases are from being unambiguously classified, so there is some quantitative measure provided (I think).
It is interesting that many people still aren’t familiar with conformal prediction. As Jessica says, it is now the topic of quite a bit of attention. I’ve been following it, and even doing some modeling, for around 7 years. I remain convinced that the primary reason so few people are familiar with it is that the literature is unnecessarily mathematical (not that that would prevent most readers of this blog). But the “simplified” or “intuitive” explanations I have read are anything but. I don’t believe the idea is really that complicated, but I have yet to see a sensible interpretation.
Dale, the question I have about Conformal prediction is, what’s the point? We have Cox’s theorem which says that Bayes is unique up to isomorphism in the systems that generalize 2 valued propositional logic, and we have Wald’s theorem that tells us that any non-Bayesian decision rule has a Bayesian decision rule which reduces the risk uniformly for every possible reality (ie. that you can ALWAYS do UNIFORMLY better by searching for and choosing an appropriate Bayesian decision rule), Conformal prediction isn’t Bayesian probability and therefore you’re guaranteed to be making higher risk decisions and doing so in a way that violates basic ideas in logic. Why would you do that?
Hasn’t Cox’s “theorem” been refuted? https://arxiv.org/pdf/1105.5450.pdf
No, only that there were hidden additional assumptions which needed to be made explicit. Kevin S Van Horne details that and addresses the Halpern counterexample in his excellent article here: https://www.sciencedirect.com/science/article/pii/S0888613X03000513