This is Jessica. If you’re looking for some light weekend reading, Weijie Su wrote a nice introduction to the need for statistical methods for large language model development and use. It doesn’t go into much detail on any specific applications, but if you’ve been wondering how to think about LLMs relative to other deep learning models or what specific problems people are developing methods for, it’s a good overview. As an object of statistical inquiry, LLMs tend to be more interesting than prior purely predictive ML models because of the inherently stochastic nature of natural language (as opposed to fixed ground truth labels), their stronger dependence on data (over relying on human feature engineering or tuning of hyperparameters), and the highly flexible ways in which they process and appear to semantically represent anything that can be expressed through text. Su argues that our best bet is to approach them as complex systems whose inner workings are not computationally reducible in the sense of being characterizable through simple interpretable rules; i.e., we should not be expecting to arrive at some singular theoretic foundation that unites the various statistical challenges that arise.
Some of the finer points he makes may be arguable, but as a summary it captures the perspective of a growing number of statisticians, computer scientists, economists, etc. who find LLMs exciting. Essentially, they present us with a landscape of inputs, outputs, and whatever latent information we can infer from their intermediate representations that we must model and do inference on if we want to understand them. In this way, they are not unlike biological systems that we can’t understand through simple closed form laws.
Something a bit cringey that becomes clearer when you see the various statistical challenges laid out like this is that sometimes they arise not just because LLMs are too complex for us to understand, but also because they are proprietary objects. E.g., once a large LLM has been trained, it’s been found that it can be more efficient to distill its knowledge into a smaller model than to train a smaller model from scratch. This motivates developers of big models to figure out ways to make their outputs resistant to distillation by competitors. It’s all just statistics I suppose, but I’d much prefer to work on problems like uncertainty quantification or watermarking outputs than how to resist sharing knowledge! Similarly, secrecy around training data curation can make it harder to theorize about dependencies between data mixtures and model capabilities.
In reading this alongside other recent takes on the state of stats in ML, it’s interesting to me that despite a growing consensus that we need to develop interpretable models to make sense of LLMs, there still seems to be a contingent of ML researchers who dismiss further integration of classical stats. For example, Su cites evaluation of LLMs as a place where we need statistically grounded methods to avoid an evaluation crisis with similarities to the replication crisis in social science, where researchers game the evaluations they present (there are various reasons to worry about this, some of which we summarized here a few years ago). But others refer to attempts to incentivize more thorough reporting of uncertainty in ML evaluation as “a weird obsession with statistics.” What’s up with that, I wonder?
From the last link:
All progress will ground to a halt and it won’t matter how many trillions of dollars are pumped in, if/when this takes hold. The last ~100 years has shown this NHST stuff is the most effective method at stifling progress ever devised. Every field stops producing real advances (but still endless papers and press releases) as it is adopted.
Think about it, why are you testing the hypothesis that two different models perform the exact same on some benchmark? We already know they are different. Its actually great to see the way benchmark results are handled as the (very) rough guide they are.
The root problem is there is both sampling/statistical error and systematic error. Unless the potential systematic error << statistical (typically the opposite is true) , requiring hyperprecise info about sampling error only serves to confuse.
But isn’t the problem that the authors are overgeneralizing based on comparisons between their method and various baselines? It’d be one thing if experiments were presented as, here, we found a couple scenarios (e.g., through messing with hyperparameters, metrics, etc.) where our method outperforms the prior SOTA. But instead conclusions in papers imply that the new method, whether its a model, training procedure, whatever, is superior and should replace whatever came before, similar to stylized facts in social science.
Something that ML seems to have going for it over social science is that cherrypicked results are more likely to be exposed because subsequent researchers will attempt to replicate what was done. But even if everything is getting reproduced eventually I still don’t see how absolving authors from having to present uncertainty to support the claims they are making. I think there’s a genuine confusion among computer scientists sometimes between wanting to consider themselves as doing science but also wanting to somehow be above using classical statistics.
Right, overgeneralization is a systematic error.
The proposed solutions of sampling error bars and statistical significance do not account for it. De facto, it functions to present a facade of rigor while leaving the (often vast) majority of uncertainty unaddressed.
Eg, here are some real-life benchmarks: https://aider.chat/docs/leaderboards/
I recently went through a few at the top and the ranks/scores do not correspond to my experience at all. But I know I am using a specific workflow, language, toolchain, prompts, etc that differs from the bench. It can only give a rough idea and we shouldn’t pretend otherwise.
I’m definitely wary about more statistical methods lending a guise of rigor where it doesn’t really exist, and obviously don’t think statistical significance testing is the answer. But I guess part of what I find confusing about dismissing attempts to hold people accountable for what they are claiming (like the crazy long NeurIPS checklist that the linked post complains about) is that in the absence of suggesting an alternative way to better align the claims and the evidence, it reads as though we’re saying all is ok as is and uncertainty can continue to be an afterthought when presenting evaluation results. But are we really ok with people presenting partially reported stats where you can’t even tell what the stupid error bars represent? I.e., if we acknowledge there’s a problem, and we don’t think better uncertainty reporting is part of the answer, what’s the alternative advice for toning down the hype?
They can hype all they want, what matters is the results when other people use it.
I’d guess very few people really read these papers. They just try things and cite the paper if it works. Eg: https://math.stackexchange.com/questions/4271692/a-possible-bug-in-a-highly-cited-paper-adam-gradient-descent
But if we want one rule, make it sharing all the data so others can analyze it how they want.
Thanks for the interesting post and link to the paper. I have been wondering about a few of these issues and not sure which papers to look at.
More specifically: what are the best practices (if such things exist yet) for comparing, on a given task, the effect of a different choice of prompt or LLM model? Since LLMs are inherently stochastic the output varies with each run making it challenging (and expensive) to compare the effect of changing the prompts and models, or finding an ‘optimal’ prompt. In all the performance comparisons I see, authors only report point estimates without any uncertainty, which in practice I have found can be quite large.
Also there are plenty of other questions related to how to best select examples for labelling, how many labels are needed… It all seems a bit of a wild west…
I’m not too familiar with what sorts of statistical frameworks people are using for this, but I share the frustration when general claims are made but there’s no attempt to demonstrate that results are robust to small prompt variations or model variations. This comes up in some of the work on LLMs for behavioral science, where you you can’t necessarily assume that changing a prompt slightly (in ways that don’t change the semantics) will lead to the same results in LLMs. Sometimes changes to a prompt that we would not consider important from a human standpoint lead to very different results from an LLM, due to things like memorization of training data or tokenization.
The system prompt also plays a huge role. Eg, the recent grok thing has nothing to do with the model per se. It is a system prompt that anyone should be customizing to maximize their own personal benchmarks:
https://github.com/xai-org/grok-prompts/commit/535aa67a6221ce4928761335a38dea8e678d8501
It just doesn’t really make sense to compare some kind of average +/- uncertainty here. Yes, I have been thinking the same about averaging humans too.
Like the “average human” does not exist, and won’t be getting your medical treatment. Real humans will, who really do differ from each other. This diversity is not an “error” to be minimized.
There is definitely some “innate” difference in capability between models though.
I should have read further long. It looks like page 11 of that paper on Uncertainty Quantification and Evaluation has some useful references too.
Jessica:
Interesting discussion. I wonder if some of the debate, such as it is, depends on framing more than anything else.
In the title of his paper that you link to at the beginning of your post, Su asks whether these language models “need” statistical foundations, but in the abstract he argues that they would “benefit” from statistical contributions.
In his post that you link to at the end, Recht objects to checklists as a form of “bureaucracy” and refers to statistics as “a bunch of arbitrary rules.”
I wonder if the implications of human language and rhetoric are pushing the two sides apart.
On one side, Su makes very reasonable arguments for the value of statistics in the development and assessment of computer language models. But then in the title he uses the word “need,” which is some form of rhetorical overkill, in the sense that large language models are computer programs (more precisely, interactions of computer programs and human operators) that already exist and serve many functions, so clearly they don’t have any absolute need for anything more. One could argue that that large language models have been developed in light of statistics (including Bayesian methods!), so in that sense they retrospectively have “needed” statistical foundations already, but I take Su’s point to be that these models would benefit from additional statistical insight; he’s not just talking about the statistical foundations that were already there.
On the other side, Recht presents a very reasonable engineering perspective that is anti-bureaucratic: we’ve already made a lot of progress and continue to do so, so don’t tell us what to do. Or, to put it more carefully, you can tell us what to do for safety or public policy reasons, but it seems like a mistake to try to restrict researchers’ freedom in the belief that this will improve research progress. This general position makes sense to me, and it is similar to many things I’ve said and written regarding science reform: I don’t want to tell people what to do, and I also don’t want criticism to be suppressed. That doesn’t mean that science-reform proposals are necessarily bad. For example, I find preregistration to be valuable (for the science, not the p-values), but I wouldn’t want it to be a requirement.
Anyway, my point is that, just as Su seems to be making a logical leap from the reasonable statement that language models should “benefit” from statistical foundations to the possible claim that these statistical foundations are “needed,” it seems to me that Recht is making a leap in the other direction from the reasonable statement that language models don’t “need” new statistical foundations to the claim that statistics is “a bunch of arbitrary rules.”
I’m sitting here saying, “Can’t we all just get along?” Here are some quick points:
– Language models have developed in light of decades of developments in the foundations of computer science, statistics, linguistics, psychology, etc.
– With that in mind, it seems reasonable that future developments in language models will benefit from new ideas in statistics.
– To flip that last point around: language models are successful enough now that I’m sure they’ll continue to be successful even in the absence of any systematic research at all, let alone new statistical foundations. I think new statistical perspectives on language models can be useful and even important; I don’t think they’re needed.
– Checklists can be useful. It depends on what’s in the checklist. Checklists of required steps can be useful (and should be held to a higher standard than optional checklists); again, it depends what’s there.
– The Neurips checklist is ridiculously detailed–and not just in its requirements about statistics. It’s absolutely nuts, a nightmare of bureaucratic red tape–and it’s required for all submissions! Ultimately, this is Neurips’s choice. If they want to make it a pain in the ass to submit papers there, and if they want to impose a bunch of stupid requirements–including all that crap about “statistical significance”–then, hey, it’s their call. People can still publish their papers on Arxiv, ICML, JMLR, etc. It just seems like a bad idea to me.
– The existence of a stupid and epically bureaucratic Neurips policy should not be taken as a sign that future developments in language models will not benefit from new ideas in statistics. A stupid checklist is a sign of a stupid checklist, and it’s an indication that there’s some sloppy thinking going around, along with some unfortunate committee dynamics. Obsessing over statistical significance and error bars (“It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of normality of errors is not verified”) is a bad idea; there’s a lot more to statistics than hypothesis testing error bars! Indeed, I think there are very solid statistical reasons for being uninterested in the endpoints of a confidence interval. And that “96%” thing, that’s just preciously stupid–why not go whole hog and say “95.4%”?
OK, I’m drifting here. The point is that we should not let our thinking be defined merely by opposition to foolish things. Much of the Nudge industry is stupid; that doesn’t mean that all of social psychology is a scam. The much-hyped beauty-and-sex-ratio research is complete crap; that doesn’t mean that evolutionary biology is useless. A Neurips committee gave bad statistical recommendations; that doesn’t mean that statistics itself is a bunch of arbitrary rules.
– Although statistics is not “a bunch of arbitrary rules,” it can be perceived that way by non-statisticians, and statistics textbooks can sometimes encourage this attitude with their rigidity. In Regression and Other Stories we try to do better; also econometrics books by Angrist, Pischke, Cunningham, etc., present pragmatic takes, but traditional statistical education still often promotes an ideology that alternates uncomfortably between very strong assumptions (simple random sampling with perfect response, ideal randomized experimentation, Poisson and binomial distributions, etc.) and equally strong statements that nothing can believed in the absence of whatever particular assumption is being focused on at the moment.
– One of the great things about computer science is its openness to different theoretical approaches. This is something that computer science shares with political science–research in theoretical foundations is itself part of the discipline–and it differs from, say, statistics and econometrics, two fields that are often hampered by conceptual rigidity. The openness of computer science to different perspectives is a good thing, and it should not be taken to imply that different theoretical frameworks (including those of Bayesian and frequentist statistics) will not be helpful in current and future research in the field.
Thanks, nicely said!