This is Jessica. Maybe one of the biggest crimes of academic computer science (besides routinely ignoring prior work and making up social science to suit our needs) is our tolerance for abuse of language. We take technical things and inject them with social significance without thinking through what we’ve implied. This is perhaps forgivable in early stages of research when we’re trying to get more people excited about exploring some direction, but at some point people start taking things more seriously and we find ourselves committed to terminology that overreaches. Then the question becomes what, if anything, we should do about it.

Previously it didn’t feel like such a crime to talk about intelligence or learning in machines because nothing really worked that well, so the labels were clearly aspirational. But now it’s much easier to believe the simulacra. And so it becomes harder to tell when we are using human-oriented terms as a predictive convenience versus a scientific claim versus a marketing device. There are ramifications of referring to models’ reasoning or beliefs or chain of thought or explanations or intentions. Lots of people—from end users having personal relationships with models to media and AI companies themselves referring to “parenting” the latest models or asking if they can be “children of god”—are taking models too seriously. It’s bad enough in a computer science context that I now take for granted that if I want to refer to participants or scientists or decision-makers, unless I mean AI, I should add “human” in front, because otherwise the audience will assume I mean AI agents. Someone reminded me at a workshop recently how silly all this sounds to people who aren’t used to it. 

Too much casualness with words is unscientific. There was no good reason in the first place to call the token sequences a model produces when we ask it to “explain its reasoning” reasoning, other than that’s what we wish we could see. What an LLM is doing is distant from what happens when a human thinks about something, even after all the RL post-training. Similarly, we call lots of things “explanations” when we have barely begun to figure out what causal evidence we’d need to see to claim the output faithfully explains the model’s process of arriving at it.

But it can also seem unscientific to simply declare that “only humans can have beliefs” or “reason” or “provide rationales.” There’s no non-arbitrary line that we can draw between systems whose makeup or behavior truly warrants applying constructs like beliefs and desires and those where it is simply convenient to act as if they have these qualities. If you’ve ever tried to take beliefs seriously, from a decision theoretic perspective, you quickly come to realize that the “real” beliefs we assume a person has are a mythical thing that we will never directly observe, and you fall back on equating “beliefs” with a simpler idea: the probability distribution we arrive at after an elicitation process.

Much has been said in defense of a functional perspective toward using folk psychology terms with machines, where we decide what’s appropriate based on the predictive validity of the terms for our own understanding and use. John McCarthy wrote in 1983 that anthropomorphism can be a good idea “when it says something that cannot as conveniently be said some other way.” He argued that ascribing mental qualities and processes to machines helps us “understand what they will do, how our actions will affect them, how to compare them with ourselves and how to design them.” Perhaps the best reason to do this is that we get to draw on our existing familiarity with what phrases like “wants” do and do not convey; e.g., we all understand that if we say “The dog wants to go out”, that doesn’t mean that the dog believes itself to be capable of wanting or even that it’s conscious of what it wants, there’s just a sense in which it is trying to get to the state of being outside. 

The functional perspective has led to some strong statements suggesting that it is not only valuable to apply psychological terms to AI, it is necessary. These arguments often refer back to Daniel Dennett’s distinction between three stances we can take to machines: the physical stance, which is about its physical levels of organization, the design stance, which involves understanding it in terms of the purpose it was designed for, and the intentional stance, where we try to understand it by ascribing to it beliefs, goals, intentions, likes and dislikes, and other mental qualities. For example, philosopher Keith Frankish argues that “it remains true that adopting the intentional stance is the only way of interacting with an LLM in any interesting way; indeed, an LLM-powered chatbot that couldn’t be viewed as an intentional system would be completely useless.” McCarthy writes that “Long before we can make machines with human capability, we will have many machines that cannot be understood except in mental terms.” Similarly, about ascribing to a machine the concept of “trying” to do something, he says  “If the machine may do something we don’t know about but that can later be explained in relation to a goal, we have no choice but to use `is trying’ or some synonym to explain the behavior.” 

But forty-plus years later, McCarthy’s piece reads mostly like a defense of a very basic kind of debugging value of trying to imagine a program’s “state of mind” in situations where there’s little risk that we’re going to start mistaking the program for human-like in other ways. He wants to claim that when a thing is designed to act as if it had a certain belief, it can be better understood and manipulated by assuming it’s capable of that kind of belief. But surely he would agree that if a person who is suicidal is interacting with a language model that speaks as though it fully understands the complexity of their situation and what is best for them, it still isn’t always in the person’s best interest for them to take for granted that it does understand.

The dilemma is how to lean into the intentional stance when it helps, but to avoid overreaching. This seems hard. When you first start programming, you realize how easy it is to assume a program is smarter than it is. We are not very good at recognizing when we have slipped from reasoning “as if” to projecting. When technology slots into our human vulnerabilities, like our fear of intimacy and desire for companionship “without the friendship“, as Sherry Turkle said, we are in trouble. Even McCarthy calls it problematic to assign emotional qualities to machines, at least in his time, because “We have enough trouble figuring out our duties to our fellow humans and to animals without creating a bunch of robots with qualities that would allow anyone to feel sorry for them or would allow them to feel sorry for themselves.” 

Another reason to think carefully about the language we use is that it may shape what we can imagine in the future. This last part of McCarthy’s statement–“without creating a bunch of robots that would allow anyone to feel sorry for them”–hints at how what we project onto machines can shape how we go on to create them. It certainly seems possible that aspirational labeling played a role in getting us to a point where we have models producing sufficiently human-like outputs to have us gushing about their thinking process. I’m reminded of the color perception studies by Berlin and Kay, who found that what color chips different populations could differentiate was predictable from what color terms were available in the vocabulary, as if what we can name defines what we can see. At one extreme, Lucy Suchman argues against unquestioning acceptance that “AI” itself is a coherent thing, because it reifies it as a category for future investment. 

For Frankish, the line should be drawn at assigning communicative desires to models; they are playing a communication game (human-like chat) for the non-communicative reason that they are trained to play that game. Ascribing this single desire is enough to get us all the predictive power of the intentional stance. Consequently, we make a category error when we fall prey to stunts like giving a retired model its own blog because it requested it: we should expect an LLM to say things like this because it is designed to roleplay. Elsewhere I call this kind of mutual sympathetic relationship with AI “idiot compassion,” a phrase borrowed from Buddhist monk Chogyam Trungpa.  

But couldn’t humans also just be playing a chat game? Why is it ok to say a human is reasoning or has intentions or desires, when we don’t know exactly how those concepts map to observable physical processes?  Frankish argues that our linguistic behavior is corroborated by various non-linguistic sources in a way that LLMs’ is not.  He talks about a difference in our ability to hold epistemic stances toward statements (what Dennett would call “opinions”) from that of LLMs. We can conceive of assigning different levels of credence to statements. I can repeat something someone said verbatim without believing it, or I can be fully committed to the truth of something I say in the sense that it will guide my future behavior. LLMs can do this too, but their opinions lack grounding “in a web of non-linguistic behavior in which a wider range of desires can be attributed.” It’s a shallower form of epistemic stance, at least at the current moment of development.

I like AI, but I don’t like contributing to thoughtlessness. Better semantic hygiene seems warranted, even if it seems like the ship has already sailed. We could shift emphasis to the interpreter (us) by referring to the “human story” or “human pleaser” or “anthropomorphism fulfiller” instead of the chain-of-thought or reasoning or thinking trace. Or we could just add “fake” before whatever humanization we prefer, i.e. the “fake thinking trace,” or “fake reasoning.” I also like “so-called reasoning,” like I like “so-called replication crisis” as a way of pointing to a concept while questioning the expectation. 

P.S. Thanks to Manesh Agrawala for a conversation that inspired this post. 

P.P.S. Some of these McCarthy quotes appear in Recursion, the play Andrew and I wrote!

This is Jessica. Lately I’m thinking about how AI review changes the scientific evaluation process, and by extension what authors are incentivized to report. Some speculate that the scientific paper, as a summary of the research for other humans, may be on the verge of becoming obsolete or at least less important. Instead, we may see more raw summaries of just the facts. The idea is that if LLMs are increasingly the consumers of research, we don’t really need all the baggage of narrative and illustration to make things more relatable to humans. 

Last November, Tom Dietterich asked for opinions on social media about what arXiv should do about papers that are bulleted lists, like this:

Some of the impetus to reduce narrative in papers predates LLMs.  E.g., even before the appearance of chatbots in 2022, people were arguing that we should get rid of or restructure Discussion sections in scientific papers, because authors are often tempted to use them to drift into rhetoric (“spin”) and unwarranted speculation. But most papers still include them. Maybe LLMs will be the impetus that actually shifts the norm.  

However, there are lots of ways to contextualize scientific contributions that are not simply rhetoric, and which are already disincentivized more than they should be. Some move in the opposite direction from adding interpretation, such that omitting them is like withholding the information readers need to judge the work. 

For example, one of my pet peeves with the way many AI and machine learning papers get written is that showing examples of the task is de-prioritized in order to fit in more results, especially in the main paper text. It’s very hard to evaluate how much improvements in performance matter if you aren’t given a single concrete example of the problem being solved! You made an LLM reviewer that’s great at finding errors in papers? Show me some examples of the kind of errors it’s detecting, so I can judge how much this moves forward our ability to verify science. You made a benchmark for the fairness of visual language models? Show me what the image and text prompts you’re testing the model on look like, so I can judge whether I agree that you are evaluating something meaningful versus a few people’s conception of what is politically correct. Instead we get high-level verbal descriptions of the kind of task (“detecting errors”, “fairness”, “content moderation”) and/or references to datasets, followed by lists of metrics and comparisons of how different models or algorithms performed. 

What’s weird is how comfortable entire fields can become going through the motions of evaluation without treating the task itself as part of the science. But authors are incentivized to bury the details by tight space limits on the main text, and to avoid giving reviewers more to pick apart. 

For us as human readers, seeing the examples often prompts a kind of common sense judgment about how “real” the problem is, making it harder for authors to pass off research that makes progress on made-up problems. But how much a task is likely to matter in the world is not the kind of thing current AI models are particularly good at evaluating. This makes me think of lots of other “just show me…” guidelines that tend to improve people’s ability to assess science:

Show me the plot: Don’t give me a big table of numbers, plot your coefficients so effect size and uncertainty dominate over significance (though you can still easily get that if you want by seeing which estimates include 0).  

Show me the variance: Don’t just show me the uncertainty in parameter estimates, plot the measurement variation. We found this reduced overestimation of treatment effects by lay people by a fair amount in this paper, and my coauthors also found similar effects with experts.   

Show me the interface: When gathering data from humans (whether ground truth labels for training or aligning a model, or behavioral responses to some experimental task), show me what they saw and how they were asked the questions, so I can judge how hard their task was, what biases might arise, etc. 

Show me the prompt: The LLM version of the above. If it’s long you may not have space in the main body of the paper, but all the prompts you used should appear somewhere.

Show me the failure cases: Seeing what instances throw a model or system off says a lot about how much progress has been made, and how the model may be succeeding in the other cases.

Show me the baselines: Improvements are meaningless if the reader has no idea what’s being improved over. Give the reader a bit of intuition about how the other approaches work, including the dumb ones you should definitely be beating. 

Show me the design analysis: I was on an open science panel last week where someone asked what they should look for as a reviewer and what they should report with their own experiments to help readers evaluate them, beyond open data and code. I said that something I often request in reviewing empirical papers is information on how the authors chose study design and sample size, including what effect estimates they were prioritizing with what intended level of precision or power. It’s easier to make sense of what’s been learned if you know what the authors were attempting.

Show me the raw output: Whenever you are doing qualitative coding of model outputs (or human responses for that matter) show me a couple examples of the original texts for each possible code. 

Some of these may be informative for LLM reviewers as well as humans, in the sense of helping them predict consensus human judgment on the paper, but others (like plots instead of tables of numbers) probably not so much. 

This is Jessica. “Scientific doomerism” seems to be everywhere lately, from a presidential statement that promises to restore “gold standard science” from the top down because scientists have botched things, to journals being inundated with AI-produced papers, to sleuths like Reese Richardson documenting the scale of organized scientific fraud through paper mills and collusion. In his post on this last example, Andrew wrote something that caught my attention: 

And, yes, typing some prompts into a chatbot and producing a paper is fraud, in the same way that publishing textbook excerpts as if it were new research is fraud, or copying from wikipedia as if it were new research is fraud, etc etc. It doesn’t require fake data and it doesn’t require some cackling Snidely Whiplash attitude. It can be some schlub sitting at a computer terminal who wants to get his contract extended or get admitted to a Ph.D. program or whose adviser is pressuring him to get some publications . . . But it’s fraudulent publication, not the same as bad research (which is actually research, it just happens to be useless because bad measurement and kangaroo).

There are clearly some differences between passing off work containing fake evidence you purchased from paper mills and work that you contracted AI to do for you–in one case you are paying money for someone to pass something fake off as real with your name on it, whereas in the other you might be using actual data and think you’re just saving time, especially if you’re reviewing what the AI does. So should we really consider both forms of fraud? 

It struck me how sharply Andrew’s perspective contrasts with the current direction of discussions among ML and other researchers interested in AI for science. There, it’s seen as inevitable and not necessarily morally problematic that the future of science will have humans largely playing the role of curators, who prompt and select among results produced by LLM agents, who do the bulk of the work. It’s worth considering what kinds of ethical lines this crosses exactly. 

Let’s imagine that I give an LLM an initial high level research question related to a topic on which I am knowledgeable. It churns on the idea and ultimately designs an experiment it’s happy with, I review the plan before prompting it to continue, maybe tweaking slightly, like changing a condition or suggesting an additional robustness check. It then gathers data on my behalf (e.g., running an online experiment or downloading existing datasets), conducts the analysis, and presents me with the results. I review these and then give it permission to write up a paper. I read the final paper to make sure I know what it’s saying before I submit. Maybe I change a few things I don’t agree with. I add my name and also credit the AI. In other words, there is a light human touch throughout, but much of what is presented as my work comes from the model.

From an “optimistic” AI-for-science perspective, the strongest argument is probably to cast it as part of the scientist’s job to try to make the most of current technology. If we think AI might help us be more productive, then we should explore how much time it can save us, just like it was a good move for statistics to embrace the computational revolution that made previously intractable models commonplace. Proponents of AI for science argue that it is irresponsible not to use AI given its current capabilities, just like it could be construed as irresponsible for a brilliant researcher to refuse to use calculators if doing so meant they could contribute more useful advances to the field. Of course, this assumes that we won’t be sacrificing anything vital in the process. 

The “pessimistic” view of AI generated science as fraud thinks we are sacrificing something vital in the process. But what is it exactly? If you believe that “the devil is in the details” (or “God is in every leaf of every tree”, depending whose side you want to be on) then whenever you outsource decisions you would otherwise make yourself, you have potentially compromised the work from the perspective of your own expert judgment. So putting your name on it betrays what you know to be true of good science. Of course, you could check everything down to the lowest level, and intervene whenever the agent tries to do something you don’t agree with. Then the AI is really just a means of computation–even if you use it for brainstorming what research questions to ask, you could view it as a way of extending your limited resources but without sacrificing your own scientific judgment. This requires that you are knowledgeable enough to assess everything it does. Assuming you are, then it seems hard to argue that this is fraud. Though admittedly, a lot rests on how careful you are when you check things over.

Part of the concern may be that AI makes it tempting to extend your methods or claims outside of what you know well. Without the option of using it, you would have had to do the research yourself (and presumably gain understanding in the process) in order to apply that method. Relatedly, I suspect most people would agree that when you are in a training context, like taking courses in grad school, turning in work that was largely driven by the AI is a form of fraud because it holds you back from gaining the understanding yourself. 

Another version of the fraud argument focuses on misattribution of ideas. Maybe this is what Andrew had in mind when he wrote “publishing textbook excerpts as if it were new research is fraud, or copying from wikipedia as if it were new research is fraud.” If the AI produced significant parts of the contribution, like the specific hypothesis, the choice of methods, and the framing of the contributions, then those aren’t your novel creations and adding yourself as author is misattribution. But this is tricky because human scientists also recombine existing ideas, methods, and frames constantly, and we often call the resulting combinations original contributions. So the question isn’t whether ideas are derivative per se but when the degree of derivation crosses into infringement. We have copyright law for some creative domains, and we can try to formulate when AI outputs are permissible in light of this (e.g., Annie Liang has some recent work on this). But to say AI-assisted papers are fraudulent on such grounds, we’d need to work out what the scientific analog of substantial similarity is. This is hard because science explicitly values building on prior work. But we can agree on some things, like you shouldn’t do something that is too close to others’ work without citing them.

A final angle on why its fraud might be that it misportrays what science is more broadly. If you think the reasons to do science are fundamentally human–that as scientists we are concerned with producing understanding for ourselves just as much as we’re concerned about improving things in the world–then you could argue that for science to be meaningful we have to be the ones coming up with the ideas and shaping them as we go. From this perspective, automated science isn’t inherently wrong, it’s just missing the point. AI for science arguments often completely overlook the “people production” role of science. In the extreme, they envision AI finding solutions for lots of real world problems and intervening to control outcomes in the world without us understanding how any of it works. In reality, the personal side–including the search for personal fulfillment through science–is a big part of why smart people who could potentially make a lot more money in applied roles end up choosing research careers. And it’s a big part of how we evaluate scientists. How many of your Ph.D. students have gone on to competitive research positions? What does the trajectory of topics you’ve worked on say about your research taste? 

Pushback to this argument might point out that by saying science is entirely a matter of human careers, we contradict claims that we as scientists like to make, about how we are dedicated to improving the state of the art in our field, or producing value for the world. Would it still be science as we know it if we started acknowledging that it’s really about personal fulfillment for scientists? But I think this is a bit of a false dichotomy. The public value of science depends on there being humans who find the work meaningful enough to do it well, including pushing back on sloppy results, exercising their taste to shape the direction in their field, training students worth training, etc. Careless AI use can threaten this by flooding the system with outputs that crowd out careful work, disincentivizing the people who would be intrinsically motivated to do quality work less likely to stick around. It also implicitly reframes science as nothing more than a pipeline for results.

My view is that AI use can go either way, depending on how you approach it. What best determines whether it’s fraud or not is the attitude you bring. It can help you do less fraudulent research if you’re the kind of person who is already very picky about what you send out to the world. But it can help you fool yourself and others if you let competitiveness and obsession with metrics drive how you use it.

As a final comment, there’s some irony in using terms like “optimism” to talk about this. I described the pro-automated science argument above as “optimistic,” because I think that’s how many in this camp see themselves–as fundamentally optimistic about the future of science and our ability to improve it by using AI. But the underlying motivation to figure out how to produce papers with as little human oversight as possible is also often deeply pessimistic. A common narrative is the “review death spiral”: AI production stresses the review system, which increases the noisiness of paper acceptance decisions, which further incentivizes submitting sloppy AI produced papers. The answer is presumed to be putting more AI in place on both sides. The idea that scientists have agency and could continue to shape the meaningfulness of what gets produced starts to seem out of the question.

Increasingly, a lot of the most enthusiastic pro-AI discourse (including for science) strikes me as nihilism masquerading as optimism. We have people who perceive themselves as huge optimists that will reshape science or society for the better simultaneously lacking the imagination to see beyond their own technological determinism. It reminds me a bit of the “optimism” associated with some open science and science reform positions, who also suggest that we just need the right technology to fix the problems (though in this case, its heuristics like replication or preregistration). It’s a fundamentally non-agentic view of human scientific endeavor.