This is Jessica. I’ve previously written some posts musing about conformal prediction as a decision aid. Part of what got me interested in it as a method for quantifying uncertainty is the tension between how on the one hand, prediction sets might seem more intuitive or concrete to decision makers (compared to providing a pseudo-probability that a prediction is correct like a softmax value), but on the other hand, it’s unclear what the typical decision-theoretic utility-maximizing agent would do with a conformal prediction set. Now I’ve written a paper reflecting on this further.
As a brief reminder of what conformal prediction is, in a classification setting, split conformal prediction takes an instance (e.g., such as some patient symptoms in a medical diagnosis example) and generates a prediction set containing a set of labels expected to contain the true label with a user defined probability 1-alpha. Typically this coverage is expected to hold marginally over the randomness in the data.
To generate prediction sets, we take some held-out labeled data (which should be distinct from the model training or test sets), and for each instance in this calibration set, we take the value of a heuristic measure of prediction uncertainty that our model provides for the true label (this could be, e.g., a softmax score from the final layer of a neural net, or a residual in a regression setting). We convert this into a non-conformity score, i.e., a score that takes larger values the less “conforming” the label is from what the model has learned. For example, the score could be as simple as 1 – softmax(y) for label y. We then calculate the 1-alpha quantile of the non-conformity scores for the true labels in the calibration set. Given some new instance, we use this 1-alpha quantile as a threshold, including in the prediction set all the labels with non-conformity scores less than the threshold.
Yifan Wu, Dawei Xie, Ziyang Guo, Andrew and I write:
Methods to quantify uncertainty in predictions from arbitrary models are in demand in high-stakes domains like medicine and finance. Conformal prediction has emerged as a popular method for producing a set of predictions with specified coverage on average, in place of a single top prediction and confidence value. However, the value of conformal prediction sets to assist human decisions remains elusive due to the murky relationship between coverage guarantees and decision makers’ goals and strategies. How should we think about conformal prediction sets as a form of decision support? We outline a decision theoretic framework for evaluating predictive uncertainty as informative signals, then contrast what can be said within this framework about idealized use of calibrated probabilities versus conformal prediction sets. Informed by prior empirical results and theories of human decision making under uncertainty, we formalize a set of possible strategies by which a decision maker might use a prediction set. We identify ways in which conformal prediction sets and post hoc predictive uncertainty quantification more broadly are in tension with common goals and needs in human-AI decision making. We give recommendations for future research in predictive uncertainty quantification to support human decision makers.
What might decision makers do with prediction sets?
In the paper, we contrast prediction uncertainty that has a well-defined use by a loss-minimizing rational decision maker (such as a calibrated probability, which the rational decision maker uses to update their beliefs then choose the action expected to minimize their loss), with conformal prediction, where the decision strategy that a risk-neutral rational decision maker should use is underspecified. Our goal is to characterize various strategies that decision makers might apply. Several of these have anecdotally come up in the literature or been mentioned by participants in studies we’ve run, but not formalized.
For example, decision makers have reported using a prediction set to verify their own judgments. If we assume the decision maker has a prior probability distribution over labels (p(Y)), then upon seeing a prediction C_hat(x), for each possibly label y, the decision maker can update their beliefs by noting whether the label is in the set, normalizing its prior probability, then multiplying it by either 1-alpha (if in set) or alpha (if not). I.e., the posterior probability for all labels in the set C_hat(x) becomes (1-alpha) * p(y)/(sum of p(y’) for all y’ in C_hat(x)). For labels not in the set, we replace (1-alpha) with alpha and normalize each label’s prior probability by the sum of probabilities of all labels not in the set.
However, some decision makers might instead use sets to constrain their decisions, such as when a doctor rules in or out certain conditions based on what labels appear in a set. The simplest strategy for this kind of decision maker would be to just divide probability equally among all labels in the set and assign zero probability to all labels not in the set. However, we can also imagine more sophisticated strategies, for example that capture the intuition that prediction sets reduce the “cognitive cost” of making a decision. Here, we can look to a rational inattention model for economics, where the decision maker’s objective is to find the action that minimizes loss that incorporates a penalty term representing the cost of information acquisition (which in this case is the attentional cost associated with considering a label).
We can also imagine variations on verify and constrain strategies where decision makers perceive labels as having some similarity relation over them. This was reported by study participants in our empirical work and figured as well as in recent work on decision-focused prediction sets. Medical conditions can be related based on how similar treatments or risk levels are. Labels for objects that appear in images can be related using semantic ontologies like the Wordnet hierarchy. Perceived similarities between labels might factor into how the decision maker uses sets or assigns value to them. For example, given some similarity function over labels, the decision maker might “transfer” probability from labels that appear in the set to similar labels that don’t.
Shortly before I planned to post this paper, Kiyani, Pappas, Roth, and Hassani posted an interesting related paper asking what kind of decision process makes prediction sets the right way to communicate uncertainty. They show that when you have a risk averse (as opposed to risk neutral) decision maker who seeks to maximize their minimum utility despite ambiguity about the data-generating distribution, then a max-min rule is an optimal strategy for using prediction sets with marginal coverage, where the decision maker chooses the action that maximizes their utility in the worst case over label outcomes. They also show, somewhat surprisingly, that prediction sets contain all of the information necessary for this kind of risk-averse decision making.
Questioning the usual restrictions on the decision maker’s knowledge
Another thing we discuss is how common reasons for involving human decision makers in the first place complicate the goal of producing valid uncertainty estimates. Namely, we are often hesitant to replace human decision makers completely with statistical models because we think they have some valuable knowledge to contribute over the information available to the model. The problem is that when we allow for the decision maker to potentially have access to some private information not available to the model, then for all we know, the calibrated uncertainty quantification we are so carefully trying to achieve may be the wrong information to provide. See for example this demonstration by Corvelo Benz and Rodriguez of how a calibrated model probability may not help the decision maker with access to private information optimize their choice of action. This tension is rarely acknowledged in theoretical and algorithmic computer science research on uncertainty quantification, so we devote some of our paper to spelling out the problem .
Ways to better align conformal prediction with what we know about human decisions
We make some recommendations for how to better align research on conformal prediction with what we know about human decision making under uncertainty. This includes advice for empirically studying how decision makers use prediction sets, such as by formulating benchmarks representing expected performance under the different strategies we characterize and comparing observed human decision performance to them to try to tease apart what people are doing. We need to be looking for ways to design uncertainty quantification despite the potential mismatch in information access between the expert and model (e.g., by multicalibrating prediction uncertainty over the model and human’s information as described here). There is also a lot we still don’t know when it comes to how people respond to different properties of prediction sets. We can do a better job of informing aspects of prediction set design with preferences we elicit or learn from expert decisions, such as for the diversity of items in a set (i.e., should seeing a set ever prompt the decision maker to think more divergently than they otherwise would?) or the shape of prediction sets in higher dimensions. More generally, it’s possible that how human decision makers are informed by prediction sets is less sensitive to coverage than the conformal literature implies, in which case we should consider instead maximizing coverage subject to other constraints. If you’ve ever studied how people use confidence intervals, you’ll know what kinds of insensitivity I’m talking about.
Why it’s hard to “price in” conformal prediction
We conclude with some thoughts on why it’s hard to appraise the value of different approaches to quantifying uncertainty, including post hoc prediction calibration methods like conformal prediction. One issue is that often the factors that drive the popularity of certain methods for quantifying uncertainty lie outside the bounds of theory. The standard bootstrap, for example, is widely used in part because it’s a simple concept that promises uncertainty quantification without much thinking, not necessarily because theorists can prove ways to improve convergence as sample size grows. Then there is the challenge of assigning value to certain forms of uncertainty that some decision makers may desire (e.g., Bayesian probability) because they don’t fit into the framework we’re working in (e.g., loss minimization; see prior post on this here). There are some things theory is hard pressed to comment on in current forms, but this doesn’t mean we should pretend they don’t exist.

I’ve read a bit of your paper, and the blog post, and tried to understand. This is way too unfamiliar and way too difficult for me, and as soon as formulas and numbers are presented, I am usually lost. I did however want to ask a question which you can disregard if you like to. I am just trying to understand a bit more, and come from things differently and way more basically.
What I thought about when reading about how a person might have some extra information or might have an additional beneficial effect on decision making was intuition. When reading a bit in your paper, it seemed to me that the extra information you were talking about is actual concrete information that could be incorporated in the AI information database (or what is the term) but just happened to not be compared to the information the person has.
I read some stuff about clinical intuition in diagnostic decision making concerning psychological disorders because it was part of my thesis for graduating. If I remember correctly, intuition in that light might be present in some experts, and they might not necessarily be consciously aware of having that information. If that makes sense, I wondered how that fits into the section about combining human and AI “available information” in decision making.
To give a concrete example, I was wondering whether it could be that in your example of the picture of the skin an expert with “good intuition” concerning this particular case might use their intuition to diagnose condition B (the correct one) and not necessarily be aware which “available information” they actually used to chose diagnosis B over A (and C, D, etc.). For instance, the picture of the skin might contain some other characteristics that makes diagnosis B more likely correct, and could be used by experts with the “correct” intuition in this case to choose B over A (or C, D, etc.) without them possible being aware that they used, or even saw, this other characteristic. The same reasoning might apply to other non-consciously used information, like age, that may unconsciously be used by experts with the “correct” intuition in this case to chose B over A (or C, D, etc.) without them truly knowing that played a major role.
Interesting question. I tend to think of intuition as knowledge based on experience but accessed implicitly/unconsciously (at least this is how it came up in a psych class I took in grad school). Sounds similar to how you are thinking about it.
In the paper, when we distinguish between the “data generating model” that the human uses to produce their prediction (e.g. of the true diagnosis) versus that used by the statistical model, both can be thought of as associating some set of features/context with different conditions. However, the set of features that the human has access to might include information that the model can’t access. This way of characterizing it is still compatible with the human not necessarily consciously understanding what information they are using. It’s just that to formalize the idea of the human having access to different info than the model, it makes sense to write it out as access to some inputs that don’t overlap with the statistical model inputs.
If you haven’t read it, I love Meehl’s book on clinical versus statistical prediction https://psycnet.apa.org/record/2006-21565-000
Quote from above: “This way of characterizing it is still compatible with the human not necessarily consciously understanding what information they are using.”
I think I understand that, I just wonder 1) if and how this implicit intuition part is accounted for in the formula/model/whatever the appropriate term, and 2) whether making a difference between the implicit intuition human information and other more explicit information (that just happens to not be present in the statistical model/computer) makes sense in your models, and more importantly in your paper and general way in which the human-computer decision making comparison is presented.
Again, I don’t know anything about models and formulas, but I reason that at a certain point it might be good to incorporate things in them that are as close to real life or a real situation as possible. In that light, I was wondering about the things concerning intuition. For me, that might be the exact point at which the comparison between humans and computers might be interesting or relevant or whatever the appropriate word is.
I have not read your paper yet, so you may discuss this. If possible, can you give a brief response to my question below?
My understanding of conformal prediction is that the output (putting aside the methods involved in obtaining this output) for a classification model is that observations are assigned to classes, with the possibility of an “uncertain” class, meaning that subject to a user-specified probability the classifications can be relied upon. What prevents this from becoming another form of binary NHST thinking? Classifications that are found within the classes are not questioned, and the uncertain classification is either abandoned or subject to further research. Instead of discrete classifications, don’t we want a continuous measure of the likelihood of each classification?
If this makes any sense, I like to think about a concrete case: medical diagnosis. It seems to me that the ultimate decision problem is to decide on a classification, including perhaps, that more testing needs to be done. But a conformal prediction either hides that problem behind a pre-chosen probability level, or leaves the clinician with the quandary of when to rely on the classification produced. I guess the question is whether the output of the conformal prediction helps with such decisions or not. Compared with a calibrated probability form a classification model, it isn’t clear to me whether this is an advance.
I’m not Jessica, so I probably can’t give you a direct answer to your question, but you might want to check out this paper which is cited and discussed in Jessica’s paper: https://arxiv.org/pdf/2502.02561
I have not fully read through this paper yet, but at a high level one of their claims is that a certain form of risk averse decision making can be accomplished by optimizing a utility function over a prediction set (and therefore you just need the prediction set, no continuous likelihood values needed).
Thanks, yes, this new Kiyani et al. paper characterizes a setting where outputting conformal prediction sets is the optimal thing to give the decision maker.
Another paper that studies a simple binary classification setting is this one by McLaughlin and Spiess: https://arxiv.org/pdf/2208.07626
They show that if you have a decision maker who changes their decision rule conditional on an algorthmic recommendation (such as when the DM sees deviating from the algorithm’s suggestion as something that carries a higher cost), then returning a set is well motivated. In this scenario, returning a set (which is the same as abstaining from predictions) reduces decision loss.
Maybe it’s just me at this stage of life, but I found the mathematics in the paper you linked bothersome. There’s a lot to wade through and I’m not sure it really clarified the important issues. I get the idea that it can be superior for an algorithm to refrain from providing a recommendation, such as having an “uncertain” category in a binary decision problem (isn’t that what conformal prediction does?). But, the idea driving the analysis in the paper is that there is a loss function when the decision maker departs from the algorithmic recommendation.
Current practice by clinicians is that there are practice “standards” and practice algorithms produced by humans: these are the result of complex combinations of historical practices, evolving medical knowledge, and a variety of quasi-regulatory processes. It is already true that clinicians face potential losses if they deviate from standard practice (both in terms of malpractice exposure and risk of non-payment from insurers). For example, if a patient is under “active surveillance” for prostate cancer, there are standard frequencies for doing biopsies (ignore for this discussion the fact that these are currently undergoing change). A clinician deviating from that standard practice faces potential losses, and these are likely to bias their decision making. The question addressed in the paper linked concerns how a machine learning algorithm may influence decision making.
It seems to me that the only difference is whether the standard or algorithmic recommendation was produced by a historical human process or by a machine process. I think that distinction may be important, but I don’t see that the paper sheds any light on that – or that it even considers that at all. Any clinical decision is made with a variety of information inputs, and I believe these inputs always exert a force on what a clinician decides. I can see dangers in that, although I don’t believe decisions would be better if clinicians were deprived of such information (there are clear cases where this isn’t true: just think about malpractice worries that cause unnecessary testing). So, I think the relevant issue is to compare the usefulness or problems with machine-based recommendations vs. human-based standards of practice. I don’t see how the mathematics in this paper clarifies or addresses this issue.
Perhaps I just got lost in the details – if so, please clarify for me.
Dale –
Many of the papers I read these days feel overly tedious so I know what you mean :-)
I think the theory in the McLaughlin and Spiess paper would apply even if the recommendation was coming from some human originating “recommendation model.” The relevant part to understanding why its better to return no prediction (or equivalently, a set containing both labels in the binary problem they consider) is Figure 2b and Proposition 2, where they show how if a decision maker anticipates some extra loss when they deviate from the algorithmic recommendation and end up being wrong, their decision threshold will be more stringent than it needs to be relative to the optimal threshold, and so the loss their decisions actually incur is higher than if they chose the loss minimizing action without the algorithmic recommendation.
Jessica
So, put more simply or crudely: doctors fearing malpractice suits may sub-optimally prescribe more tests/treatments?
Yes. I have heard anecdotally from medical researchers that sometimes introducing an AI into a hospital directs attention to ineffeciencies in the current (human) decision process, such as overprescribing tests at unnecessary cost, but healthcare organizations tend to be very squeamish about letting the introduction of a statistical model lead to less “safe” decisions than the conventional process did, even if it means knowingly incurring unnecessary loss.
Cutting the plane of the particular choice of quantity of interest (and whether one chooses a Frequentist- or Bayesian-motivated quantity of interest), which is problem dependent, the most important property of an estimator for final-stage decision-making is some non-vacuous accounting of the epistemic uncertainty. This is particularly critical for decision-making over high-dimensional objects, such as LLMs, because it may not be obvious to human end-users that the distributions have shifted. There are loss functions [1] that make explicit the epistemic uncertainty (cf., “6.2.3. Second-order uncertainty” and in contrast to, e.g., Eq. 8 in Bengs et al. 2023, which only takes into account the distance to the decision-boundary and is thus insufficient a priori) and corresponding methods for partitioning distributions based on the epistemic uncertainty. Statistical assurances for decision-making from methods such as Risk Averse Calibration (Kiyani et al. 2025) are largely nullified in practice when not taking the epistemic uncertainty into account, at least in high-dimensional settings.
[1] For example, with an SDM loss, the penalty becomes higher for instances with lower Similarities (depth-matches into training—a signal of epistemic uncertainty) as a consequence of a change of base vis-a-vis a standard Cross-Entropy loss. Bengs et al. 2023 suggests there may not be such loss functions; in fact, there are such loss functions, and they work well in practice.
Thanks, are there papers you suggest on this?
Similarity-Distance-Magnitude Universal Verification. https://arxiv.org/abs/2502.20167
See Section 6.1 (p. 21-22) for the key takeaway with respect to my earlier post. The code, replication scripts, and all of the models and outputs (approximately 45 GB uncompressed) are available linked in the public GitHub repo (Footnote 1).