Ilan Strauss and Tim O’Reilly ask:
Why don’t ML and LLM model evaluations report uncertainty? Rarely see an interval of some kind.
– Because the models are too big (LLMs)?
– Or because their ML metrics (Accuracy, recall, precision) are assumed to be sufficient for taking into account uncertainty in predictions stemming from the model’s training data, i.e. can the model generalize / predict new data points?
They continue:
LLM behavior is also inherently uncertain. The model’s responses are highly sensitive to factors like the query, hyperparameters, and context, all of which introduce variability in a model’s outputs. . . LLMs seem to be computationally deterministic in their outputs (even if practical stuff complicates this): Given the same input and conditions, the model should generate the same probabilities for the next token. The variability we see in outputs stems largely from the sampling methods applied on top of these probabilities, such as top-k sampling or temperature sampling. These techniques introduces randomness, producing different outputs for the same input. But even without this sampling layer, uncertainty should persist in LLM evaluation results because it’s impractical to test all possible model input-output combinations. . . .
Calculating and showing model uncertainty usually comes by providing an interval . . . So, why the omission by OpenAI of uncertainty from most of its model evaluations? Maybe computer scientists aren’t always familiar with common statistical practice . . . the leading AI textbook by Stuart Russell and Peter Norvig (4th edition) [has] an entire chapter on “Quantifying Uncertainty”, but devoted largely to uncertainty facing AI in the external environment. . . .
I don’t have any answer of my own to the question, “Why don’t machine learning and large language model evaluations report uncertainty?”, for the simple reason that I don’t know enough about machine learning and large language model evaluations in the first place. I imagine they do report uncertainty in some settings.
My general recommendation to people running machine learning models is to replicate using different starting points. This won’t capture all uncertainty, not even all statistical uncertainty—you can see this by considering a simple example such as linear least squares, which converges to the same solution no matter where you start it, so in that case my try-different-starting-values trick won’t get you anywhere at all—; rather, I think of this as a way to get an approximate lower bound on Monte Carlo uncertainty of your output. To get more than that, I think you need some explicit or implicit modeling of the data collection process. An explicit model goes into a likelihood function which goes into a Bayesian analysis which produces uncertainty. An implicit model could be instantiated by repeating the computation using different datasets as obtained by cross validation or bootstrapping.
I’m aware of two approaches aimed at expressing uncertainty in such models: https://re.express/index.html (which I discovered from a comment on this blog) and conformal prediction (which has been discussed several times on this blog). But I share interest and some puzzlement expressed in the question. Many of these models result in probabilities for classifications: so there is uncertainty, but not uncertainty about the uncertainty. Since there is not usually a formal error model in many ML methods, there may not be an obvious or established way to produce uncertainty estimates. For example, it is easy to generate confidence intervals from logistic regression models, but not so much from tree-based classification models or neural nets. One approach I’ve seen for establishing uncertainty about the importance of factors (rather than uncertainty in the predictions themselves) is to replace each factor with a random variable to quantify the importance of each factor. I kind of like that approach, but I’m not sure why the next step isn’t taken to quantify the resulting prediction uncertainty.
Good question, important in my mind. I look forward to learning more.
There have been various calls over the years for ML evaluations to better report uncertainty. We talked about this in our “worst of both worlds” paper a few years back. Hidden degrees of freedom on which point estimates depend, like unreported computational budgets and other hyperparameters, come up in sections on model selection and evaluation and communicating claims.
https://arxiv.org/pdf/2203.06498
That’s a nice article!
On a related note, I had been excited by the prospect of using uncertainty estimation methods (ensemble/Bayesian/epistemic NNs : https://doi.org/10.1007/s10462-023-10562-9) as a straightforward way of estimating reproducibility in ML-based science. But then I discovered that it may not be as straightforward https://doi.org/10.1007/s10462-023-10562-9. (a finding that using several ensembles is equivalent to using a single larger model). And I’m curious about the latest in this subfield.
Here’s the correct second link: https://arxiv.org/abs/2202.06985
You are right that “replicate using different starting points” will only capture a small fraction of the uncertainty in model evaluation. A few years ago, we did a systematic study of the variance of machine-learning benchmarks ( https://proceedings.mlsys.org/paper_files/paper/2021/hash/0184b0cd3cfb185989f858a1d9f5c1eb-Abstract.html ) and we found that the dominant factor was the sampling noise on the test set.
Looking at the spread of the evaluations on the various folds in a cross-validation is thus a good gauge of uncertainty of a benchmark. Modern machine-learning practice frowns upon cross-validation because 1) it induces compute costs, 2) some of the aspects of the pipeline are not automated and thus cannot be cross-validated. I however think that this is bad practice.
I have a working paper in which we use my R package idealstan to estimate LLM input as indicators in a measurement framework. You can then derive uncertainty for a latent trait that they estimate–though this is primarily useful for an annotation task.
Could you please share it.
“Calculating and showing model uncertainty usually comes by providing an interval . . . So, why the omission by OpenAI of uncertainty from most of its model evaluations?”
Interval around what? Incentive really matters here. The goal of the LLM is to produce an output that makes the user think they are interacting with an intelligent lifeform. When I asked an LLM how to use the notoriously confusing public transportation system in Milan to get from one place to another, it gave me a plausible-sounding answer that turned out to be absolutely useless. But since I was originally satisfied with the answer and moved on to something else, it pretty much achieved what it set out to do. How would it know otherwise?
If we imagine an LLM moving spatially around a map of all the information it knows, we can envision that there are many answers to a query that might potentially satisfy the user scattered around the information space. Meanwhile, the most accurate answer would be a banal return of a wikipedia-like regurgitation of the question after rewording (an LLM response with no “creativity”).
How is uncertainty defined when certainty itself is an invalid construct?
More a response to the original post, but LLMs and machine learning are completly different beasts. Talking about them in the same breath seems nuts (to me, but I’m me).
Machine learning is gradient decent search in extremely high dimensional spaces with explicit domain knowledge and/or calculations.
LLMs do random matching/instantiation of sequences of undefined tokens with no domain knowledge or calculation.
Personally, I have opinions on these things. I _hate_ the term “machine learning” because it’s not learning anything, but at least it’s serious computer science. But since we know what LLMs do under the hood, that they produce garbage outputs randomly is completely expected (and in mathematical principle unmodifiable) behavior.
This strikes me as a very peculiar set of opinions and I have a hard time understanding where you’re coming from. Let me make some statements that hopefully help bring that out and perhaps you can explain what you mean (if willing!).
First, talking about LLMs and machine learning in the same breath is widespread and commonplace since LLMs are a particular kind of ML model. Research on artificial neural networks (of which the Transformer architecture LLMs are based on are a subset) has always been considered a part of machine learning. Indeed, it is explicitly mentioned as such by Arthur Samuel in his foundational paper exploring ML strategies for checkers.
Second, ML is a broad term which characterises a heterogenous collection of approaches to developing computer programs. These approaches are typically unified by the use of some automated procedure to implement the data-driven adjustment of model parameters with respect to an objective. Gradient descent, high-dimensional spaces, and explicit domain knowledge are all optional features of particular ML setups, but do not define it. Further, the first two of these ingredients are central to building LLMs, so I don’t follow why you mention them in the context of explaining why ML and LLMs are “completely different beasts”.
Third, there are many ways to talk about what LLMs do. Inarguably, they compute probability distributions over sequence items (tokens) via large numbers of tensor operations. Beyond this, there is a lot of disagreement over how best to characterise what they do. Personally, I find your characterisations perplexing.
In what sense is the computing of sequences of tokens “random”? It seems to me that the sequences of tokens are not random at all, in the sense that they are selected with respect to (several, sequentially implemented) objectives. What do you have in mind here?
Then, in what sense is it “matching/instantiation” as opposed to generation? To be clear, I’m familiar with the empirical literature showing that LLMs produce text with less originality than human authors (i.e., a larger proportion of multiword phrases are present in the linguistic corpus). However, this is both a) a relative effect (LLMs still routinely produce novel n-grams for large values of n), and b) less true for some models than others (i.e., it is a property of specific training regimes, not a fundamental property of the model class).
I also wonder what you mean when you call tokens “undefined”. Do you mean they are not assigned a unitary symbolic meaning (a la good old fashioned AI systems)? Or do you mean that they are ungrounded (i.e., not coupled in the right kind of way to the worldly things they purport to represent)?
I work at OpenAI. I don’t think there’s any deep answer – it’s a mix of bad historical convention and laziness and trying to write for a worldwide audience.
Internally we of course measure error bars on our LLM evaluation results. But as with all errors, there can be nuance and confusion over what they mean. E.g., if you want to put error bars on a GPQA diamond result, in which you have, say:
– have 3 topics
– have 150 questions
– have 10 samples per question
– have 1 model trained
Then even a notion of “sample size” can be ambiguous, and you need to make sure any error bars you publish is appropriately communicating your model of clustered errors, etc.
Someone told me of perhaps the greatest logician of the 20th century “… WVO Quine that scientific problems are not closed problems, all scientific theories are under-determined (by both theory and data), and therefore the degree of uncertainty in scientific argument is insuperable, logically insuperable (he was a logician)”. AI cannot solve open problems but for closed problems they might be able to provide uncertainty bounds —eg jackknifing/bootstrapping as Andrew has mentioned with Monte Carlo lower bounds (very costly since GPT 4 cost 300 million dollars to test — equivalent to firing a rocket).
I once attended a sales pitch by an ML company whose twist was that their framework would store not only the weight, but also the certainty. They would do a linear regression on a data set like everyone, but they’d remember that point X1, Y1 was in the data set (so they have 100% confidence there). If you ask for point X2, Y2, you’d get not only an answer but also a confidence value depending on how close X2,Y2 is to the training data.
I assume that this kind of thing only works for basic ML that is not very computationally expensive, but I only saw the sales pitch, not the implementation details.
P.S. I mention linear regression as the easy-to-understand example, but their overall product was for image recognition/categorization.
I’ve puzzled how anyone could do quality control on LLM’s. As the question noted, “[…] because it’s impractical to test all possible model input-output combinations. . . .” since the input space is a sequence of 100’s, or 1000’s, or 10000’s of tokens chosen from a set of around 2^16 possible tokens, we’re talking about input sets of cardinality around 2^16^100, 2^16^1000, or 2^16^10000. So the smallest of those is a 482 digit number, after you’ve tested a trillion (10^12) examples your percentage coverage has 468 zeroes after the decimal point. What do those trillion samples tell you about nearby sequences in the input set? I’m not sure how statistics works in this situation.
For LLMs, sdm estimators address, and effectively resolve, the uncertainty problem: https://arxiv.org/abs/2502.20167
I am not aware of any other classes of methods that would reliably work over high-dimensional models.
There are (at least) two separate questions here. One concerns reporting confidence intervals on expected error rates for a predictive model. For simpler ML models (e.g., boosted trees, SVMs, etc.), many ML papers report 95% confidence intervals using the holdout method (simple test set). As Gael points out, these only capture uncertainty due to sampling error of the test set (and some sources of measurement error in the inputs and outputs). With cross-validation, you can approximate the uncertainty due to the training set. For LLMs, simple test set methods can be applied for problems, such as multiple choice questions, for which it is easy to determine whether the output matches the desired output. With language, one usually wants to match at the semantic level (and possibly at a pragmatic level). This is more challenging, but methods based on textual entailment have been applied with some success.
The second question is to give a prediction interval on the output for a specific input query that captures both the uncertainty due to training data that influence the output and uncertainty due to measurement error in the inputs and outputs. This is studied in the literature on selective classification, where the goal is to have the ML system abstain if it is highly uncertain about the predicted output. For simpler ML models, cross-validation and ensemble methods can be applied. For SVMs, there is nice work on efficiently computing the leave-one-out cross-validation variance. Note that conformal prediction quantifies the uncertainty of a single predictive model and does not consider uncertainty due to training set sampling variation.
For LLMs, computing prediction intervals shares all of the problems of semantic output evaluation as well as the computational challenge that fitting an ensemble of LLMs is infeasible. If we are content with just assessing the aleatoric uncertainty, we can draw multiple samples from the LLM to construct a prediction set. People have also applied conformal prediction to this case. Again, textual entailment methods can be applied to incorporate semantic similarity rather than exact matches.
A couple of relevant papers:
Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy. Nature, 625–630(July 2023). https://doi.org/10.1038/s41586-024-07421-0
Vashurin, R., (et a.) 2024. Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph. ArXiv, 2406.15627(v1). https://arxiv.org/abs/2406.15627
My knowledge is about one year out of date; there is probably more recent relevant work.
Neither Farquhar et al. 2024 nor Vashurin et al. 2024 compare to the SOTA approaches publicly available before those two papers (see my papers from 2022 and earlier, as well as the no-code, on-device LLM software from Reexpress AI released in 2023) for the models with non-identifiable parameters (e.g., LLMs). Suffice it to say, the problem is much, much more complicated than those two papers would suggest (w.r.t. relevant quantities of interest, robustness to distribution- and label- shifts, ability to examine the reference classes, controlling for the effective sample sizes, etc.), but addressable, as described here, which is the evolution of the aforementioned SOTA approaches, incorporating uncertainty-aware verification and interpretability-by-exemplar as intrinsic properties of the LLMs, themselves (i.e., for both train- and test-time search/compute/generation): “Similarity-Distance-Magnitude Universal Verification”: https://arxiv.org/abs/2502.20167.