This is Jessica. In my last post about the tension between the necessity that we ascribe human folk psychological concepts like thinking and reasoning to machines and the problems that arise when we overinterpret them, I briefly mentioned a defense I sometimes hear for anthropomorphic terms. The defense goes like this: We can’t protest the application of words like reasoning or belief or desire or understanding to AI because humans are also not always trustworthy reporters of the latent states that these terms refer to. And if humans can’t demonstrate full awareness of their reasoning process or their beliefs or their understanding or their intentions, why should we expect AI to? So, the argument goes, it is asymmetric to require complete faithfulness of LLM traces to some latent state. That would be setting a higher bar than we currently use with humans.
This line of reasoning takes two things that may resemble each other on the surface (e.g., what humans report their reasoning or belief or intention to be, and what machines report) which we have no solid reason to believe are produced by similar processes, and says they cannot be distinguished because we expect both to be reported with loss.
It’s bad logic. But more than that, it occurs to me that by doing this we indirectly deny the value of our subjective, non-verbal experience. It’s true we may not be able to faithfully report how we reasoned or what exactly we desired or intended or understood. But part of the reason we have terms for these things is because we perceive a distinction between more genuine and less genuine versions of them within ourselves. These qualitative distinctions are how we come to believe that reasoning and belief and intention and desire exist in the first place: because we can perceive them as being authentically present to different degrees in different situations, even if sometimes we seem to be mimicking the real thing rather than really doing it. It’s what licenses researchers to keep trying to study these things in people despite the difficulty of getting an unbiased read. I’m not saying we shouldn’t look for analogous processes in machines. But we should acknowledge that the referent for these terms is grounded in subjective experience. Among humans that has worked fine, because we assume ourselves to share an understanding of what it’s like to have an inner life.
Andrew recently analogized chatbots with being in autopilot in conversations that occur in a meeting, where if you have a rich enough memory bank of observations related to what the speaker is saying, you can spout appropriate conjectures and feedback without having to exert much conscious effort. But I think fake thinking extends to much more than just being on autopilot in conversation. It can seep into all of our decisions, whether about research ideas or methods or life decisions like what job to have or where to live. It’s very tempting to live by patterns, even when it’s not what you feel you actually want.
Our perceptions that there are more real and more fake versions of our own thinking or reasoning or belief or desire is what makes us feel more “alive” or “awake” in some situations over others. It connects us to the present. Only when we try harder to do these things authentically or recognize the motives that are actually driving us (despite the explanations we might want to assume) do we feel like we are really living, rather than just performing some role. And so our ability to perceive the difference seems intimately connected to the process of understanding ourselves.
From this angle, it is unfortunate how much of the AI rhetoric we’ve come to take for granted (at least in machine learning) — i.e., AI as scientist, AI as decision-maker, etc.— enacts this implicit move of equating human and machine processes by their outputs, and consequently subtly devaluing the role of our internal reality in giving these terms meaning. We take examples from the few domains where language can fully capture reasoning–math, coding–and we reduce all reasoning or thinking or intention to what can be made manifest.
Maybe this helps explain why there’s so much emphasis lately in certain circles (including tech) on being “high agency.” We become more insistent about shaping our external world as we lose appreciation for our internal one.
I think you have stretched the issues with defining human “reasoning” too far in trying to defend human traits of “emotion,” “subjective,” and “nonverbal” experience. I have no problems identifying those three terms as uniquely human while making the argument you protest: equating human and machine processes by their outputs (although I don’t equate them, but I do think the comparable outputs, combined with difficulty explaining how human reasoning works, is a valid comparison to make). I don’t believe machines have emotion or subjective experience, at least as far as we humans understand these terms. I’m not nearly as sure about “reasoning.” I don’t see why ascribing human ideas about reasoning to machines undermines the idea of what it means to be human. That seems like a modern “domino theory” of a slippery slope that can’t be avoided.
I can’t conceive of ascribing emotion or subjective concept of soul to a machine – at least not based on my current understanding and experience. But I find attempts to distinguish between machine and human reasoning fall short, except for the obvious reality that a machine and an organic being are not the same thing. Why is it so hard to separate reasoning from emotion? If these are really similar processes, then it should be easier to distinguish between human and machine reasoning. People keep trying, but I’m not convinced.
Hi Dale,
I’m not saying people should stop using terms like reasoning or thinking with machines if it’s helpful (which my last post discussed more). I’m just saying that how we come to understand what these terms mean in the first place depends on a subjective dimension. It’s a source of awkwardness when we apply them to AI, because we’re trying to separate some aspects of them out from others that have been essential to our belief in them. So it’s more a philosophical commentary than call to action. Though I do think there are consequences of the rhetoric common in ML research these days in the sense of putting all the emphasis on outputs.
I think that there is indeed a manner of functioning that closely resembles chatbot utterances. It’s deeper than just spitting out boilerplate in a meeting. These are occasions where a person blurts out something before they realize they are saying it. In these cases, it seems, there is little supervisory control. That is the case for current chatbots.
We only know about chatbots from their utterances. If we want to compare with people, we should start with their utterances. People’s utterances are shaped by many things in addition to the idea they are putting forth: one’s attitude toward what he is saying, how one wants to be perceived by the listener, how appropriate the utterance will be, how he wants to listener to behave, how one imagines he will feel after making the utterance, one’s status or cultural standing in relation to the listener, and no doubt many more factors. Some of these factors may be expressed in the utterance by prosody.
As long as chatbots don’t operate according to some theory of mind, and have no way to apply such filters and supervision, they are not operating like humans no matter how smoothly their text may read. In their absence, there is no reason to suppose that “awareness”, “consciousness”, or emotions have anything to do the chatbot output.
“Some” theory of mind? “Any” theory of mind? What about the theory that consciousness is a delusion, and the mind is nothing more than a neural network processing sensory inputs and producing mechanical outputs? The chatbots have all kinds of filters and supervisory algorithms that were baked in, just like our filters and supervision are certainly learned to some extent and also have some level of “baked-in” that we’re working hard to quantify but haven’t manage to quantify yet.
If I’m understanding Jessica, the objection to judging only outputs is that we don’t judge ourselves by outputs alone. That is true, but whether or not it is real or just an epiphenomenon of the neural process that generated the output (rather than being an input to the process) is still up in the air as a hypothesis, I think.
We have treated humans as machines so long it seems only fair to treat machines as humans in the same contexts. The idea of the rational economic actor, the amoral lawyer, the faceless bureaucrat – all of these can be achieved more perfectly with AI than they ever could by actual people. The real question is whether the AI itself will be effectively insulted by the process to the degree humans should have been.
Quote from the blog post: “Maybe this helps explain why there’s so much emphasis lately in certain circles (including tech) on being “high agency.” We become more insistent about shaping our external world as we lose appreciation for our internal one.”
“High agency”
F#ck me
What a term
It makes me squirm
It’s like something a reckless, type A personality, CEO, manager-type person would say
It’s the kind of thing that might create many problems just before they simply walk away
“Tune into the frequency”
Perhaps that’s what’s best for me
Largely let things flow
Partly adjust, and let it go
Yep.
> But more than that, it occurs to me that by doing this we indirectly deny the value of our subjective, non-verbal experience. It’s true we may not be able to faithfully report how we reasoned or what exactly we desired or intended or understood.
Good line. LLMs aside, this makes me think about communicating with people at work. The reality is both kinds of thinking (verbal and non-verbal) are everywhere, but the pressure is to always present plans as if they were fully thought through.
I’m happy with the first thing (the reasoning for the plan is somebody sez so), but the second is kind of annoying (the reasoning for the plan doesn’t line up with the plan).