This is Jessica. Lately I’m thinking about how AI review changes the scientific evaluation process, and by extension what authors are incentivized to report. Some speculate that the scientific paper, as a summary of the research for other humans, may be on the verge of becoming obsolete or at least less important. Instead, we may see more raw summaries of just the facts. The idea is that if LLMs are increasingly the consumers of research, we don’t really need all the baggage of narrative and illustration to make things more relatable to humans.
Last November, Tom Dietterich asked for opinions on social media about what arXiv should do about papers that are bulleted lists, like this:
Some of the impetus to reduce narrative in papers predates LLMs. E.g., even before the appearance of chatbots in 2022, people were arguing that we should get rid of or restructure Discussion sections in scientific papers, because authors are often tempted to use them to drift into rhetoric (“spin”) and unwarranted speculation. But most papers still include them. Maybe LLMs will be the impetus that actually shifts the norm.
However, there are lots of ways to contextualize scientific contributions that are not simply rhetoric, and which are already disincentivized more than they should be. Some move in the opposite direction from adding interpretation, such that omitting them is like withholding the information readers need to judge the work.
For example, one of my pet peeves with the way many AI and machine learning papers get written is that showing examples of the task is de-prioritized in order to fit in more results, especially in the main paper text. It’s very hard to evaluate how much improvements in performance matter if you aren’t given a single concrete example of the problem being solved! You made an LLM reviewer that’s great at finding errors in papers? Show me some examples of the kind of errors it’s detecting, so I can judge how much this moves forward our ability to verify science. You made a benchmark for the fairness of visual language models? Show me what the image and text prompts you’re testing the model on look like, so I can judge whether I agree that you are evaluating something meaningful versus a few people’s conception of what is politically correct. Instead we get high-level verbal descriptions of the kind of task (“detecting errors”, “fairness”, “content moderation”) and/or references to datasets, followed by lists of metrics and comparisons of how different models or algorithms performed.
What’s weird is how comfortable entire fields can become going through the motions of evaluation without treating the task itself as part of the science. But authors are incentivized to bury the details by tight space limits on the main text, and to avoid giving reviewers more to pick apart.
For us as human readers, seeing the examples often prompts a kind of common sense judgment about how “real” the problem is, making it harder for authors to pass off research that makes progress on made-up problems. But how much a task is likely to matter in the world is not the kind of thing current AI models are particularly good at evaluating. This makes me think of lots of other “just show me…” guidelines that tend to improve people’s ability to assess science:
Show me the plot: Don’t give me a big table of numbers, plot your coefficients so effect size and uncertainty dominate over significance (though you can still easily get that if you want by seeing which estimates include 0).
Show me the variance: Don’t just show me the uncertainty in parameter estimates, plot the measurement variation. We found this reduced overestimation of treatment effects by lay people by a fair amount in this paper, and my coauthors also found similar effects with experts.
Show me the interface: When gathering data from humans (whether ground truth labels for training or aligning a model, or behavioral responses to some experimental task), show me what they saw and how they were asked the questions, so I can judge how hard their task was, what biases might arise, etc.
Show me the prompt: The LLM version of the above. If it’s long you may not have space in the main body of the paper, but all the prompts you used should appear somewhere.
Show me the failure cases: Seeing what instances throw a model or system off says a lot about how much progress has been made, and how the model may be succeeding in the other cases.
Show me the baselines: Improvements are meaningless if the reader has no idea what’s being improved over. Give the reader a bit of intuition about how the other approaches work, including the dumb ones you should definitely be beating.
Show me the design analysis: I was on an open science panel last week where someone asked what they should look for as a reviewer and what they should report with their own experiments to help readers evaluate them, beyond open data and code. I said that something I often request in reviewing empirical papers is information on how the authors chose study design and sample size, including what effect estimates they were prioritizing with what intended level of precision or power. It’s easier to make sense of what’s been learned if you know what the authors were attempting.
Show me the raw output: Whenever you are doing qualitative coding of model outputs (or human responses for that matter) show me a couple examples of the original texts for each possible code.
Some of these may be informative for LLM reviewers as well as humans, in the sense of helping them predict consensus human judgment on the paper, but others (like plots instead of tables of numbers) probably not so much.
Agreed on all counts. I think you wrote a reporting guideline recently about this kind of research transparency?
Building tools to scaffold this kind of thing with plugins and extensions, IMO would be helpful. The Blank Page/Canvas syndrome is really problematic in scholarly communication. These days, I work from a set of reporting guidelines towards my outline and rough draft. I couldn’t imagine doing this anyway else now. Using a hypertext notebook like Roam Research makes the work even easier.
“The idea is that if LLMs are increasingly the consumers of research, we don’t really need all the baggage of narrative and illustration to make things more relatable to humans.”
Baggage?? This is what gets me about this type of thing. Science, at least for most scientists, is about understanding. You can’t get the AI to understand for you.
In mathematics, there has been for the past few years a discussion about formalizing proofs in a computer language called Lean. Lean verifies that the proof is correct, but it is not a substitute for understanding.
It’s also like the old idea that code is written to be read by humans. We should all write terrible undocumented spaghetti code if all we care about is execution by the computer. If we want other people to understand our work and be able to build on it and generate more understanding, it should be written for humans.
>It’s also like the old idea that code is written to be read by humans. We should all write terrible undocumented spaghetti code if all we care about is execution by the computer. If we want other people to understand our work and be able to build on it and generate more understanding, it should be written for humans.
In recent work on AI-for-science, this comes up as a tension between science as control versus science as understanding. The question is whether, now that AI is (in some ways) more powerful than human scientists, we can let science focus primarily on its post-Enlightenment goals of prediction and direct intervention (or control) to improve outcomes in the world, or if we need to still try to preserve the (traditionally human) goal of understanding. See e.g. this https://www.science.org/doi/10.1126/science.aec7650
A lot of the arguments I see for using AI to evaluate science seem to be assuming that we can’t get rid of humans entirely, because we still need them to judge novelty and signficance, but now we can fully separate out many other parts of review (reproducibility, robustness, etc.) and give them to AI, then just summarize for the human what the AI found. This paper, for example, seems to be arguing not that we drop the human-understandable narrative, but that becomes more of an interface we spin up as needed, and the real artifact that gets submitted is optimized for AI: https://arxiv.org/pdf/2604.24658
The purpose of prediction and control for scientists is better understanding, though. The sociologist and philosopher who wrote that first article you linked don’t seem to understand this point. No physicist cares about better prediction and control when it comes to electromagnetism, because for them the field is solved. The people who do want that in that field are engineers. Physicists e.g measure the magnetic moment of the muon to many decimal places because that still results in more understanding. Once that problem is solved to the extent that electromagnetism is, physicists will stop and engineers will take over.
There is also a lot of “secondary literature” in science whose purpose is systematizing and simplifying the results gained from primary literature (an example of this is review articles, and there is a physics journal explicitly for this [Reviews of Modern Physics]). Is the proposal that humans will be reduced to writing only secondary literature? Even this doesn’t quite follow, I think.
To take an analogy, computers became better at chess than humans in 1997. This has not stopped humans from playing it, because it is fun. A human even came up with a creative strategy to beat a computer by taking advantage of its weaknesses (https://arstechnica.com/information-technology/2023/02/man-beats-machine-at-go-in-human-victory-over-ai/) The fun side of science cannot be discounted.
I agree. I’d also add that prediction is constrained by the current normative views in science, e.g.., what is important predict, and what to use as potential predictors. One key (if rare) contribution of human scientists is to question or even overturn the normative paradigm. Currently it seems impossible for AI models to do this. Over time scientists learn their current theories are inadequate and that tension becomes unbearable. It’s hard to imagine AI scientists doing the same and in fact it seems that incrementally better predictions could elongate the pre-revolutionary period of science.
On the point about whether we’re on a trajectory towards giving up science to AI or whether LLM’s should be considered the consumers of research. That might be a valid trajectory in computer science or statistics, but how is that is going to work in experimental science? An issue is how AI cracks through the straightjacket of currently accumulated knowledge to develop novel directions. That’s discussed in a recent paper which aligns with RNM’s point: “Artificial Intelligence Tools Expand Scientists’ Impact but Contract Science’s Focus” in which evidence is presented that AI tends to automate established fields rather than exploring new ones: ( https://arxiv.org/abs/2412.07727 )
There are experimental fields where AI is or will be very useful, e.g. cross correlating phenotypes associated with rare diseases with the vast library of current therapeutics, for drug repurposing. The evolutionary nature of scientific progress doesn’t seem so amenable to AI approaches though. If AI had been around in the 1990’s early 2000’s and attempts at gene therapies using viral vectors, no doubt it would have done an excellent job of finding patterns in the (failed) attempts at clinical interventions. But successful gene therapies developed in a completely different direction largely due to obscure findings by a couple of researchers of curious repeat sequences in bacterial DNA and then attempts to understand their functions. Crispr technology developed into a viable therapeutic approach to treating (and curing possibly) rare genetic diseases through some serendipitous findings along with innovative approaches by experimental scientist and important inputs from the human genome project.
I haven’t been following crispr, but looking at the early data it was clear it primarily worked by being selectively toxic (killing cells containing a target sequence, mutants survive) and would therefore be unsuitable for any in vivo use.
I see the stocks I was watching at the time went:
EDIT
2016: ~$30
2021: ~$100 (high)
2026: ~ $3.00
CRSP
2016: ~$20
2021: ~$200 (high)
2026: ~ $50
NTLA
2016: ~$20
2021: ~$200 (high)
2026: ~ $10
NB: SPY increased by ~4x over this period, and CRSP did release one ex vivo treatment (Casgevy). There are still no in vivo treatments.
So could current AI have identified this issue early on for people who don’t/can’t interpret the data for themselves? I’d kind of doubt it without custom post-training or extensive (100k token) guardrails/docs trying to enforce skepticism.
The main problem is its too trusting of its training data, so will too easily default into parroting the author’s interpretations. You would be fighting an uphill battle.
My point was that the evolutionary nature of experimental science makes it unlikely that AI would become some sort of dominant approach (in experimental science) or that AI should be considered the consumers of research. The Crispr story was just an example of the way that scientific advance often relies on serendipity and innovation that breaks out of the existing knowledge base. I wasn’t expecting a discussion on stock prices of med tech!
That’s not the case. The Crispr system in bacteria has been develped as a technology that amongst other things does seem suitable for in vivo use. There’s a recent example of successful treatment of an infant with CPS enzyme deficiency which involved direct injection of Crispr components encapsulated in lipid nanoparticles.
Since these treatments can be specific to a single patient (since different patients may have different gene variants to correct and so a different Crispr system has to be designed for each) they’re hugely expensive, but the FDA has just approved an approach whereby patients with different pathogenic variants in a gene can be grouped in trials targeting that gene and this is likely to bring costs way down.
Looks like a “very low dose” and no confirmation of any “editing”. I also found papers where patients with this condition survive into adulthood. It appears the initial hyperammonemic episode (in the first weeks of life) is the usual CoD, which this patient had already survived.
So check back in a few years I guess.
https://pmc.ncbi.nlm.nih.gov/articles/PMC12713542/
I came here to say the same thing as Anonymous, so in an attempt to say something new:
If you want to improve outcomes in the world, you must generalise your results beyond the context you have investigated. (Unless the world of interest is very, very small, which most commentators agree our world isn’t).
How do we do that without an understanding (i.e. theory)? My impression is that philosophers of science agree that we don’t.
If I want people to drink more of my brand cola, I can design some study that shows the benefits which may be related to variables highly controlled in my design. I can show how cola relates to A, B, and C and conclude or recommend certain things perhaps.
Then, people might buy my brand of cola, or some people might recommend my brand of cola. In this scenario, I don’t know whether having people buy my brand of cola is considered to be an example of improving an outcome, but I think it might be the case that real science, actual facts, and actual understanding might not be necessary for “science” to play a certain role in certain things.
Take the case of the blog post: perhaps a flawed study may have had who knows what influence and consequences. You might just need a head start in things when you are a big company. Just one paper that paves the way for certain measures or actions that will buy you a few years of making some other moves. It doesn’t even matter that much when a paper gets retracted after 5 or more years, or whether some replication project finds some other results in 10 years time, for certain things to have a function.
“Take the case of the blog post: perhaps a flawed study may have had who knows what influence and consequences. You might just need a head start in things when you are a big company.”
Take the case of the blog post dated april 30 about a retracted study I should have written…
Return of the GOTO command!
To address one minor point. The discussion section has traditionally been where the authors get to go a bit wild and tell you why their paper is important. Its also where non-experts (like myself, in most fields) go first to see if the rest of the paper is worth wading through, so I am all for keeping them.
Also, it is unclear to me what good eliminating discussions sections would accomplish. LLMs can’t judge the significance of a result. To convey it if you are writing for an LLM (which then regurgitates it to the larger public), you have to spin the LLM. Results don’t interpret themselves.
Quote from above: “The discussion section has traditionally been where the authors get to go a bit wild and tell you why their paper is important.”
I see you are familiar with the classic Bem (2002) chapter titled “Writing the emprical journal article”! Some quotes regarding the discussion section from that chapter:
“What inferences can be drawn from the findings? These inferences may be at a level quite close to the data or may involve considerable abstraction, perhaps to the level of a larger theory regarding, say, emotion or sex differences. What are the theoretical, practical, or even the political implications of the results?”
“The hourglass shape of an article implies that your final words should be broad general statements of near-cosmic significance, not precious details of interest only to psychologist.”
“The discussion section also includes a consideration of questions that remain unanswered or that have been raised by the study itself, along with suggestions for the kinds of research that would help to answer them. In fact, suggesting further research is probably the most common way of ending a research report.”
Classic Bem! (also see “Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect.”)
Bem might have been right about discussion sections and wrong about precognition.
And what about writing an article and planning in general? Here are two more Bem (2002) quotes:
“There are two possible articles you can write: (1) the article you planned to write when you designed your study or (2) the article that makes the most sense now that you have seen the results. They are rarely the same, and the correct
answer is (2).”
“For example, if you discover an unanticipated sex difference in your data, you will want to determine if others have reported a similar sex difference or findings that might explain it. If you consider this finding important, discuss sex differences and the pertinent literature in the introduction. If you consider it to be only a peripheral finding, then postpone a discussion of sex differences until the discussion section.”
I very much appreciate your reports from the “front lines”, as you help me understand why anyone would think turning over thinking and decision making to statistical pattern aggregators would be a good idea. In particular, it sounds like there are people who genuinely believe that science has, in some fundamental sense, been “solved” such that the important thing now is just to provide data to the statistical aggregator which which then make an optimal decision for us.
My own perspective is that science is an activity done with the primary aim of helping people understand ourselves and our universe. This activity, like many other activities we engage in (art, commerce, agriculture, tinkering, etc.) often has the nice side-effect of producing new tools, which can be very convenient. But especially pertinent to science is that those new tools often enable us to develop, expand, or revise that understanding. New tools and new knowledge can each force us to revise our understanding of ourselves and our world, often in ways that could not have been anticipated beforehand.
But even if one does not share my perspective on the purpose of science, it is the case that the way in which human cultures understand the world tends to evolve in complex nonlinear ways over time. One way to put it is that understanding is not confined to the convex hull enclosed by our current theories/data. A consequence of this is that statistical summaries of the past will not be sufficient to generalize to future understanding. It’s not just that, as the cliche goes, science expands the boundaries of human understanding. Science also expands the dimensions of the space in which our understanding is embedded.
So on the one hand, I oppose turning science over to AI because it fundamentally misunderstands what science is about. But on the other hand, I oppose turning science over to AI because it is the wrong tool for the job–modern AI methods are statistical models and those models cannot generalize to new dimensions and therefore will fail to capture important ways in which science advances. In making this claim, I don’t mean that humans have some inherently superior ability to expand our understanding into new dimensions–I’m open to the idea that there are other, potentially automated, approaches that would better enable discoveries of new dimensions of knowledge, it’s just that modern AI tools can’t do that and are therefore unsuitable to carry on science on their own.
PS: I should add that I am in complete agreement with you on (a) the requirement for openness regarding methods and measurement; and (b) with the critiques you summarize about discussion sections! Contra Michael Murphy’s comment above, when reading a paper I first look at the methods and then at the results. I’d rather try to build my own understanding of what the paper is doing and why before I engage with the authors’ opinions on that matter—that way, I put myself in a better position to learn and critique both the paper and my own thinking.