This post is by Phil.
Some time ago I wrote some thoughts about “Neuromancer” ( https://statmodeling.stat.columbia.edu/2025/06/12/what-does-neuromancer-have-to-teach-us-about-the-role-of-ai-is-society/ ), which features two kinds of artificial intelligence, one of which seems like it could be realized with a Large Language Model, i.e. we could pretty much make it today. The other is something more powerful, an artificial general intelligence that not only has computational power but also imagination and desires. I think it’s an open question whether an LLM can have genuine desires (and even a genuine imagination) as opposed to being able to pretend that it does. Also an open question whether that distinction even makes sense to talk about.
I’ve read some other fiction within the past few months that has also given me things to think about, AI-wise.
First there was Feet of Clay, by Terry Pratchett. Pratchett writes lightweight, fun, but generally forgettable fantasy novels. I mentioned that book in an earlier post, https://statmodeling.stat.columbia.edu/2026/01/21/what-a-coincidence-what-a-coincidence/ , because it uses a rare plot device that happened to crop up in the very next book that I read. But I mention it now for a different reason: in the book there are golems (an animated, artificial humanoid in Jewish folklore created entirely from inanimate matter, such as clay or mud) that are treated pretty much like robots. A golem’s operating system is written on a piece of paper contained in its head. In the book, Golems are treated like we treat industrial robots or Roombas or similar: they are given simple, repetitive tasks at which they work, sometimes day and night. Nobody feels bad about using them however they want, because the golems have no emotions. Or do they? In the book some golems get together and create a golem of their own, and give it instructions that are…well, basically they are trying to create something more human. Of course, the fact that they desire to do such a thing suggests that they are not in fact emotionless objects.
Well, I just read another Pratchett book, “Thief of Time”. (Spoilers follow. Stop reading here if you want to read this book and be surprised.) This book has beings called ‘auditors’ who are responsible for maintaining order in the universe. They are described as being nearly emotionless except for hating disorder. To them, humans pretty much personify disorder so I think they could be said to hate humans. To better understand humans so they can learn to control us better, some of the auditors create human bodies for themselves and occupy them…and, uh oh, with the bodies come emotions. They get hungry, they can feel pain, things taste good or taste bad, etc. As they strive to satisfy their bodies’ desires, they start to act more and more like humans. They want things.
I mention this here because it touches on something I wonder about AIs, or at least LLMs: can they have desires? Certainly they can be told to _pretend_ they do — one could prompt an LLM to pretend that it wishes to take over the world, for example — but would it _really_ “want” to take over the world? Would it want anything at all?
Thinking about those kinds of questions, I realized that I don’t understand human emotions and sensations either. I don’t see how a bunch of computer circuits can be made to feel pain, but I also don’t understand how a bunch of nerves and neurons can feel pain either. I can understand how either one can respond to stimuli — if the temperature at this point exceeds such-and-such a temperature, fire these muscles — but I’m talking about the _sensation_ of pain. How does that arise? And is there something about a computer that works with voltages on a chip that prevents it from being able to have that sensation? Do nerves and brains somehow allow a sensation that literally cannot be duplicated in silico?
Sadly, Thief of Time did not answer any of those questions for me. But it did get me thinking about them, so I guess that’s something.
This post is by Phil
And you thought Pratchett was lightweight? Come now.
I concur that TP is, at least to me, not terrifically forgettable, since there’s quite a few of his wry jokes that are implanted in my memory
Most frequently quoted by far: “Scientists have calculated that the chances of something so patently absurd actually existing are millions to one.
But magicians have calculated that million-to-one chances crop up nine times out of ten.”
in Mort
Nobody understands this, but the philosophers have given it names like “the hard problem of consciousness” and the theory that machines CAN duplicate human intelligence is called “functionalism”.
https://www.ontology.co/essays/cocchiarella-ai-system.pdf
Sentences like “The third is based on a double reflexive abstraction on the intentional content of the self by means of a double nominaliza-
tion” make me long for the clarity of someone like Richard Feynman. I do not disdain all subject-specific jargon, which can save a lot of verbiage, but sometimes I think it’s used as some sort of signaling device that the writer is in the right kind of club, and I get that feeling here. Perhaps I’ll seek out a for-the-layman discussion of the same issue.
Phil, thanks for the thinking prompt. It got me thinking.
Re “I can understand how either one can respond to stimuli” … here is an answer. A winning short story, I highly recommend.
Let’s hope my ‘teller’ acts as per…
“Rounding Corrections” by Sandra Haynes
“Weeperfile 0000000001
09/26/2017 11:58 pm location 1874
acct# ending xxxxxxxx4580
Transaction: deposit cash $38.00
Ending balance $-3.28
“Observed: Middle age white female
“Response: … ”
…
[NO SPOILERS!]
…
https://gizmodo.com/read-the-into-the-black-contests-winning-story-set-in-1822338909?IR=T
“UMass financial analyst wins big with short story
by Emma Kemp For the Bulletin
February 8, 2018
https://amherstbulletin.com/2018/02/08/sandra-haynes-15258645/
“Generally forgettable”?! Outrageous!
You’re going to regret that “lightweight” dig at Pratchett :-)
But anyway, there’s all sorts of philosophical issues which can be debated endlessly, and AI/LLM’s have made them popular with pundits for the moment. It’s similar to when science was figuring out basic biology in an earlier era, and that cultural ferment was the background to _Frankenstein_.
I’m a 100% materialist, and I’d say the sensation of pain is something an AI could be made to have in theory. But it’s not currently understood how to do it. There’s all sorts of wild stuff in neurology, such as e.g. “phantom limb”, where someone who has had a limb amputated sometimes feels as if it’s still there. How does that happen? It gets even weirder, as sometimes the phantom limb will itch, and so there’s techniques to scratch a phantom itch on a limb which no longer exists. And this is just a taste.
There’s a lot of science fiction written about these ideas, but the “pop culture” version tends to be pretty bad.
Phil
You perfectly described my feelings (assuming I have feelings). While I generally don’t believe AI can experience emotion, other than pretending to, I’m at a loss explaining what exactly human emotion is. This is not to equate AI and human behavior – I still want to defend humans as possessing some sort of sentience or ethics that AI does not. But I can’t claim that the difference is based on any real understanding or any clear ability to articulate the difference. Instead, I am adopting the view that I can claim there is a difference and AI – thus far – cannot claim we are the same. As long as I am able to, I will try to protect the human roles to make judgements and be accountable for these. Because if we give up those roles, I’m not sure what we are left with.
You know, I think it’s about time I finally read Blindsight.
Personally, I don’t think the notion of consciousness makes much sense. If a machine passes some kind of super-turing test, then you either have to grant it consciousness or admit that the notion is irrelevant. If your neighbor was a good man and a decent human being, what difference would it make if you opened up his skull when he died and found a computer inside?
That said, as far as I can see LLMs do NOT behave like humans if you work with them for any length of time (like over 15 mins). They remember nothing, they repeat themselves, and etc. So I’m not sure the question arises. Dawkins (for example) is just an old fool looking for something to write about to make him seem relevant again.
Michael:
I disagree with your last sentence. Dawkins is old (no problem there, it’s happening to me too!) and he is in some dimensions a fool (as are we all!), but I think he’ll remain relevant for awhile, just based on a his demonstrated ability to push people’s buttons. I’m guessing that he’s ranting ignorantly about LLMs for the same reason that he’s ranting ignorantly about politics: he has strong feelings and doesn’t mind saying things that people might think are stupid.
Why does Dawkins want to do that? The simplest answer is that he’s had success with this strategy. Go back to his early books: The Selfish Gene, The Blind Watchmaker, etc., which display a mixture of insight and a willingness to make claims that go a bit beyond logic and evidence. Dawkins’s method of operation, for better or worse, is to take an idea, push it as far as it will go, express it in a way that might offend people, and not back down. This was more interesting when the subject was genetics in the 1970s than when it’s politics or chatbots in the 2020s, but it’s the same general approach.
I’ll echo the “who are you calling lightweight and forgettable” sentiment, and point out that Pratchett’s golems also distinguish between task instructions and system prompts (the “chem”), which is characteristically insightful.
I’ve been conversing about AI benchmarks – and an idea that came up was that once AI has robotic interfaces, maybe the benchmark should have to do with the ability to not just control the robot, but perform self-maintenance. In that case, it needs ‘pain’ (damage) and hunger (need for material resources to do maintenance) and fatigue (need for energy to do maintenance).
Sure, it needs to be able to sense its state in various ways and to respond appropriately. But a self-maintaining robot does not need to experience ‘pain’ or ‘fatigue’ in the way that we feel pain and fatigue. An automated vacuum cleaner can recognize that its battery is getting low and can go to its charging pad to recharge, but I don’t think (?) you would claim that it feels ‘fatigue’.