More specifically, he says:
ChatGPT can do deep, involved reasoning. It has the context capacity to do that.
I [Bob] think that human language is what is known as “AI complete”. To be good at language, you have to be intelligent, because language is about the world and context. You can’t do what ChatGPT does ignorant of the world or be unable to plan. . . .
Humans also generally produce output one word at a time in spoken language. In writing we can plan and go back and revise. We can do a little planning on the fly, but not nearly as much. To me, this was the biggest open problem in computational linguistics—it’s what my job talk was about in 1989 and now it’s basically a solved problem from the engineering if not scientific perspective.
I [Bob] am not saying there’s no limitations to using the LLM architecture—it doesn’t have any long- or really medium-term memory. I’m just saying it can’t do what it does now without some kind of “intelligence”. If you try to define intelligence more tightly, you either rule out humans or you somehow say that only human meat can be intelligent.
I told Bob that take on this might be controversial, even among computer scientists, and he replied:
Of course. Everything’s controversial among academics . . .
My [Bob’s] position is hardly novel. It’s the take of everyone I know who understands the tech (of course, that’s a no-true-Scotsman argument), including this paper from Microsoft Research. I do think if you have studied cognitive science, philosophy of language, and philosophy of mind, studied language modeling, studied psycholinguistics, have some inkling of natural language compositional semantics and lexical semantics, and you understand crowdsourcing with human feedback, then you’re much more likely to come to the same conclusion as me. If you’re just shooting from the hip without having thought deeply about meaning and how to frame it or how humans process language a subword component at a time, then of course the behavior seems “impossible”. Everyone seems to have confused it with cutting-and-pasting search results, which is not at all what it’s doing.
I’m not saying it’s equivalent to a human, just that whatever it’s doing is a form of general intelligence. What it’s truly lacking is longer term memory. That means there are things humans can do that it really is incapable of doing in its present form. But that’s not because it’s a “dumb machine”. We’re just “dumb meat” viewed from that perspective (unless you want to get all spiritual and say we have a soul of some kind that matters).
Bob also recommends this paper from Google and this one from OpenAI, and he continues:
There’s a ton of work on scaling laws now and what people are seeing is emergent behavior at certain model sizes. As in like 1% performance for 3B parameters, then 95% performance for 6B parameters kind of thing. But nobody knows why this is happening or where.
The capacity of these models is quite high, including the representation of words, representation of positions, etc. It’s generting one word at a time, but the structrure is an incredibly rich time series with literally billions of parameters.
The background here is that I’ve been reading what Thomas Basbøll has been writing on chatbots and the teaching of writing (a topic of interest to me, because I teach writing as part of my Communicating Data and Statistics course), and he recommended a long article by Giulio Alessandrini, Brad Klee, and Stephen Wolfram entitled “What Is ChatGPT Doing . . . and Why Does It Work?”
I really liked Alessandrini et al.’s article. It was at the right level for me, stepping through the following topics:
It’s Just Adding One Word at a Time
Where Do the Probabilities Come From?
What Is a Model?
Models for Human-Like Tasks
Neural Nets
Machine Learning, and the Training of Neural Nets
The Practice and Lore of Neural Net Training
“Surely a Network That’s Big Enough Can Do Anything!”
The Concept of Embeddings
Inside ChatGPT
The Training of ChatGPT
Beyond Basic Training
What Really Lets ChatGPT Work?
Semantic Grammar and the Power of Computational Language
So . . . What Is ChatGPT Doing, and Why Does It Work?
Alessandrini et al.’s article has lots of examples, graphs, and code, and I get the impression that they’re actively trying to figure out what’s going on. They get into some interesting general issues; for example,
One might have thought that for every particular kind of task one would need a different architecture of neural net. But what’s been found is that the same architecture often seems to work even for apparently quite different tasks. At some level this reminds one of the idea of universal computation . . . but I think it’s more a reflection of the fact that the tasks we’re typically trying to get neural nets to do are “human-like” ones—and neural nets can capture quite general “human-like processes”.
In earlier days of neural nets, there tended to be the idea that one should “make the neural net do as little as possible”. For example, in converting speech to text it was thought that one should first analyze the audio of the speech, break it into phonemes, etc. But what was found is that—at least for “human-like tasks”—it’s usually better just to try to train the neural net on the “end-to-end problem”, letting it “discover” the necessary intermediate features, encodings, etc. for itself. . . .
That’s not to say that there are no “structuring ideas” that are relevant for neural nets. Thus, for example, having 2D arrays of neurons with local connections seems at least very useful in the early stages of processing images. And having patterns of connectivity that concentrate on “looking back in sequences” seems useful . . . in dealing with things like human language, for example in ChatGPT.
They also talk about the choices involved in tuning the algorithms—always an important topic in statistics and machine learning—so, all in all, I think a good starting point before getting into the technical articles that Bob pointed us to above. I pointed Bob to the Alessandrini et al. tutorial and his reaction was that it “seriously under-emphasizes the attention model in the transform and the alignment post-training. It’s the latter that took GPT-3 to ChatGPT, and it’s a huge difference.”
That’s the problem with sending a pop science article to an expert: the expert will latch on to some imperfection. The same thing happens to me when people send me popular articles on Bayesian statistics or American politics or whatever: I can’t help focusing on the flaws. Anyway, I still like the Alessandrini et al. article, I guess more so when supplemented with Bob’s comments.
P.S. Also I told Bob I still don’t get how generating one word at a time can tell the program to create a sonnet in the style of whoever. I just don’t get how “Please give me a sonnet . . .” will lead to a completion that has sonnet form. Bob replied:
Turn this around. Do you know how you write sentences without planning them all out word by word ahead of time? Language is hard, but we do all kinds of planning in this same way. Think about how you navigate from home to work. You don’t plan out a route step by step then execute it, you make a very general plan (‘ride my bke’ or ‘take the subway’), take a step toward that goal, then repeat until you get to work. As you get to each part of the task (unlock the bike, carry the bike outside, put the kickstand up, get on the bike, etc.) are all easily cued by what you did last, so it barely requires any thought at all. ChatGPT does the same thing with language. ChatGPT does a ton of computation on your query before starting to generate answers. It absolutely does a kind of “planning” in advance and as the MS paper shows, you can coach it to do better planning by asking it to share its plans. It does this all with its attention model. And it maintains several rich, parallel representations of how language gets generated.
Do you know how you understand language one subword component at a time? Human brains have *very slow* clock cycles, but very *high bandwidth* associative reasoning. We are very good at guessing what’s going to come next (though not nearly as good as GPT—it’s ability at this task is far beyond human ability) and very good at piecing together meaning from hints (too good in many ways as we jump to a lot of false associations and bad conclusions). We are terrible at logic and planning compared to “that looks similar to something I’ve seen before”.
I think everyone who’s thought deeply about language realizes it has evolved to make these tasks tractable. People can rap and write epic poems on the fly because there’s a form that we can follow and one step follows the next when you have a simple bigger goal. So the people who know the underlying architectures, but say “oh language is easy, I’m not impressed by ChatGPT” are focusing on this aspect of language. Where ChatGPT falls down is with long chains of logical reasoning. You have to coax it to do that by telling it to. Then it can do it in a limited way with guidance, but it’s basic architecture doesn’t support good long-term planning for language. If you want GPT to write a book, you can’t prompt it with “write a book”. Instead, you can say “please outline a book for me”, then you can go over the outline and have it instantiate as you go. At least that’s how people are currently using GPT to generate novels.
I asked Aki Vehtari about this, and Aki pointed out that there are a few zillion sonnets on the internet already.
Regarding the general question, “How does the chatbot do X?”, where X is anything other than “put together a long string of words that looks like something that could’ve been written by a human” (so, the question could be, “How does the chatbot write a sonnet” or “How does ChatGPT go from ‘just guessing next word’ to solving computational problems, like calculating weekly menu constrained by number of calories?”), Bob replied:
This is unknown. We’ve basically created human-level or better language ability (though not human or better ability to connect language to the world) and we know the entire architecture down to the bit level and still don’t know exactly why it works. My take and the take of many others is that it has a huge capacity in its representation of words and its representation of context and the behavior is emergent from that. It’s learned to model the world and how it works because it needs that information to be as good as it is at language.
Technically, it’s a huge mixture model of 16 different “attention heads”, each of which is itself a huge neural network and each of which pay attention to a different form of being coherent. Each of these is a contextual model with access to the previous 5K or so words (8K subword tokens).
Part of the story is that the relevant information is in the training set (lots of sonnets, lots of diet plans, etc.); the mysterious other part is how it knows from your query what piece of relevant information to use. I still don’t understand how it can know what to do here, but I guess that for now I’ll just have to accept that the program works but I don’t understand how. Millions of people drive cars without understanding at any level how cars work, right? I basically understand how cars work, but there’d be no way I could build one from scratch.
Links to the Google and OpenAI papers are the same (i.e., “Attention is All You Need,” which came from Google). Perhaps this is this the one Bob had in-mind from OpenAI? https://arxiv.org/abs/2005.14165
Link fixed; thanks.
The link to the paper from Open AI leads to the same paper from the people at Google. Any chance you could update the link?
Much of human reasoning is essentially truth-preserving, right? If you put true statements in, you’ll get truth statements out, with either certainty (deductive reasoning) or a high probability (inductive reasoning).
Is there any reason to believe that LLMs are truth-preserving in this way? E.g., If you trained a model on only verifiable / consensus true statements, would it be guaranteed to output only statements that are also true?
It seems to me that there’s nothing about the architecture of current LLMs (GPT, LaMDA, Palm) that would create this guarantee. You could train it on nothing but true statements, and it could still output false statements.
This is precisely the aspect of LLMs that make so-called “hallucinations” possible, and it seems like a Real Big Limitation. There’s something fundamentally different that humans do that leads to consistently truth-preserving reasoning, when we are reasoning well (i.e., not confused, drunk, etc.). Once we can figure-out how to supplement LLMs with that capacity, then we’ll really be getting somewhere.
I do think it is possible with a hybrid symbolic/associative architecture. But that’s not what we have right now. We will eventually, to be sure. And at that point, there will be much stronger reasons to be comfortable with the claim that the models are intelligent.
No, most of human reasoning is not truth preserving, unless you take that as the definition of “reasoning”. In which case, most of what people do is not reasoning. Most human behavior is governed by relatively fast associative reasoning. Humans are notoriously bad (in both speed and accuracy) at logical chains of reasoning. The human neural clock cycle is very slow (let’s say 100 Hz), which limits the number of sequential logical steps that can be carried out. Much of what looks like reasoning to us is just precompiled chains of reasoning, not newly worked out.
Humans presumably have a fully neural architecture, and systems like GPT have clearly emergent symbolic reasoning capability (it can reason with variables instead of numbers and take multiple chains of reasoning, for example). So I don’t think we’re going to need a “symbolic layer” that’s explicit.
Thank for the thoughtful reply, Bob. I guess I am inclined to believe that reasoning requires operations performed over semantic, truth-evaluable content. LLMs seem not to care about truth values, in the sense that the computations that transform inputs to outputs don’t depend (in any clear or direct way) on the truth value of either.
Maybe this isn’t too different from humans. But then maybe most human behavior doesn’t require intelligence, it just requires adaptive habits. The distinctive human activity that tends to get us to new and interesting places with respect to knowledge is that behavior that requires intelligent, truth-preserving reasoning processes (along with creativity and luck).
Maybe you can get all of that truth-preserving stuff without expicit symbols, through emergent associative processing. Philosophers and cognitive scientists have been debating about that for several decades. But you do need it, in any case.
I think you’re getting at something when you talk about “most human behavior doesn’t require intelligence”.
I mean, sure there’s some form of intelligence in using a fork to pick up a piece of sausage, or whatever, and pretty much everyone has that… But what makes human intelligence different from say Raven intelligence is the robust ability to “reason” from some facts and some principles to an answer that is correct or approximately so about the real world through theories or models of the world. That is, in a novel situation the underlying principles together with the facts and some effort result in a correct conclusion. This is for example the kind of stuff we do when we take a physics class and use mechanics principles to figure out what would happen in a vacuum or if you added an electrostatic charge to an object or changed the material a thing was made out of, or in a biology class where you figure out what might happen in the presence of a drug that blocked the action of a cellular receptor or in chemistry if you replaced an atom in a compound with another atom…
I 100% believe that there are MANY people out there (possibly up to maybe 50-70%) who just don’t have that ability. They just respond to the most recent thing with whatever the pattern is they have for how to respond to that thing but their ability to do novel calculations is extraordinarily limited. This is why predatory businesses make a fortune off payday loans and insurance for your toaster, and automotive repair scams, and telephone scams, and why disinformation campaigns are successful to convince people that lizards dressed up as people operate the banks and hollywood studios and how baby eaters run pizza parlors as fronts for cults in washington DC and whatever else craziness there is in the news these days.
You know how sometimes you’re thinking about something and find that you’ve driven home without really noticing the red and green traffic lights, or which route you are taking or whatever? I have come to the conclusion that a large number of people operate like that kind of driving except with nearly 100% of everything they do. When forced to “think through” stuff they just punt and hope for the best.
All the evidence seems to point to 25 year olds having no better than 6th grade math ability typically. When asked to add or subtract or multiply various numbers they can carry those calculations out. When asked to figure out from a description of a real world scenario like say the size of a room and the number of coats of paint to be applied… how many gallons of paint will be needed… they punt to some general estimate “probably 5 gallons would be enough” because that number is historically enough for anything they’ve done in the past. Ask them something like how to paint the entire interior of a warehouse sized building and they’re lost. They can’t form models and they can’t reason with them.
Daniel:
I experienced that not-paying-attention-to-things-while-driving thing one day about 40 years ago when driving to work. It scared the hell out of me and I’ve always been careful about that since. Sometimes, though, this happens to me while biking. Not often, but every once in a while.
Daniel Lakeland must be dealing with a very different class of 25 year olds than the grad students and postdocs I deal with on a day to day basis! I’m pretty sure I could do the kind of things Daniel’s talking about in junior high. Certainly my nephew could do this kind of stuff really well at age 10, because I was quizzing him on this kind of thing (he’s pretty good at math—I could explain the summation of the first n integers to him at about the same age and he got the proof after I grouped them in the obvious way for him).
I think what’s really going on is that everyone, and I do mean everyone without some kind of serious learning disability, is doing some mix of quick, associative reasoning and introspective “thinking.” And what happens is that as we learn to do something, it goes from requiring conscious attention to being automatic. When you first started driving, it was a lot of effort and required a lot of conscious thought. Same as when you first learned calculus and tried to compute derivatives. You can carry on a conversation while driving, but you’ll find it gets cut off when a car swerves in front of you and you need to put more attention into driving. This is the kind of thing people are talking about when they say it takes 10K hours (5 full-time work years) to “learn” a craft. A lot of that’s composing simple actions into precomposed compound actions that you can apply with associative reasoning. This is why people who are good at something are also typically fast at it. For example, I’ve never met a good, but slow programmer, where I’m not at all using speed as a measure of quality.
My point was just that ChatGPT can do the same kind of thing we call “thinking.” That is, it can apply basic principles working forward. It can reason through syllogisms, it can chain logical connections, it can come up with novel connections and associations it hasn’t seen before. All the things that when people do them are called “intelligent” or “creative”.
Bob.
I did a bunch of analysis of standardized testing in schools from 3rd through 12th grade about a year ago. What I found was that in math testing students ability grew a lot between about 3rd and 6th grade, and then tended to decline between 6th and 8th and then bump up by 12th… But the average math score on CA state tests at 12th grade was substantially less than the 5th grade math score of my son (who admittedly tests above 99%tile and is in an accelerated math program this year learning what’s normally taught to 11th grader) and only marginally better than the math score of the average 6th grader. If you said that standardized math testing score averages were “flat after 6th grade” you wouldn’t really be that far off.
When I talk about typical 25 year olds I am talking about really *typical* ones not the ones on postdocs at the Flatiron institute or whatever. Pick a jury pool and look at the 20-30 year olds in that pool who have been out of school for at least 3 years.
Look at subreddits for construction or welding or retail workers or whatever. Look at the people who clean houses or do childcare for 5 year olds or are there to answer phones and schedule appointments for a dental office or whatever.
Having worked in construction litigation and forensic engineering you’d be surprised how often stuff happens where people just cut through the floor joists holding up floors so they can route their water pipe or whatever. They operate on the “I’ve been told to put my pipe here, this thing is in the way so I will remove it” level.
Daniel Lakeland wrote: “possibly up to maybe 50-70%”.
This is most obvious in math where each sentence needs to rigorously follow from what came before. My estimate is 98%. Of course, I don’t have hard data for that number. But, I have a lot of experience.
However, most humans do seem to be better than the AIs at not saying things that are false.
David
Most people indeed: like the guy who wrote the book on dishonesty using fraudulent data.
Still, I suspect if we could collect a large database of statement made by people and by ChatGPT, the former would have a larger proportion of truthful statements. Of course, that is now irrelevant since it will be difficult to tell whether the source of future statements are human or from an AI.
Coincidentally, I’ve been reading “Surfaces and Essences: Analogy as the Fuel and Fire of Thinking” by Douglas Hofstadter and Emmanuel Sander. I wonder how adept the AIs are at the type of analogy making/use that the book describes.
Daniel –
> the robust ability to “reason” from some facts and some principles to an answer that is correct or approximately so about the real world through theories or models of the world. That is, in a novel situation the underlying principles together with the facts and some effort result in a correct conclusion.
> I 100% believe that there are MANY people out there (possibly up to maybe 50-70%) who just don’t have that ability.
I think this is probably an overestimate – especially since it seems to be an estimate directly translated from your alook at standardized testing in math.
First, just because 50%-70% of test-takers might might fail to demonstrate such ability on a math test (among that 50%-70% of test takers, did they ALL fail ALL the questions that required that ABILITY, or did some of them demonstrate that ability on some questions but not succeed enough to demonstrate the ability uniformly?), giving people a test is not a 100% accurate way to test a given ability across other contexts.
But even more so, standardized tesf scores on the ability to “reason” as you describe on math problems, even if it were 100% reliable and 100% valid (which it isn’t) to test math ability, certainly wouldn’t directly translate 1 to 1 across all other contexts.
I think folks at this site tend to over-generalize about math- and statistics-related issues because in general your abilities are so far on the scale you have a very particular perspective.
I’m quite sure I know a lot of people who might even uniformly fail to reason as you describe on ALL math-related test problems or even on testing across a crowru of domains but who demonstrate that reasoning ability in other contexts.
This discussion about what percent of people are capable of doing what kind of reasoning is bizarre. There is a difference between the “capability” of doing such reasoning and the “practice” of doing so. I think most humans are capable of such reasoning – and, perhaps ravens are as well. I agree with Daniel regarding the fact that humans so often don’t exhibit such reasoning, and I believe it has served commercial interests to encourage humans to use intuitive reactions rather than conceptual reasoning (and they’ve gotten quite good at that). The example of driving without thinking is a bit different I think. My terrible golf swing is even more terrible if I think while swinging. Our body (and brain) perform better at many tasks when we act via intuition – provided that such intuition has been developed through practice. Even with practice, however, I think it is hard to rely on intuition to solve most math, physics, or biology problems. These require the use of mental effort (what I believe is referred to System 2 rather than System 1 – the latter being the intuitive mode of thinking).
Most human activity does not require intelligence – we’d be incapacitated if it did. But some activities do require it, and there are many areas where intelligence should be used, but seems not to be. I won’t venture those percentages but they probably differ between the graduate students that Bob teaches and the students Daniel is referring to (although the “invisible Gorilla” is capable of fooling both audiences, myself included). What is most interesting to think about is how AI might change human behavior. Will it encourage more or less use of this conceptual reasoning (which may or may not be uniquely human)?
> I think most humans are capable of such reasoning
I used to think that I could teach anyone math. I have actually tried with several people over the years, Most were incapable of thinking about one thing long enough to build the concept in their brain. Maybe if you had some way of motivating them (grades in school?), you could force them to make the effort. But, it isn’t clear that they would ever succeed. One person did manage to do it, but things that I understood in seconds when I was in college took them hours or days with my coaching.
When I was a teaching assistant in graduate school for a large calculus class, the students complained that the exam questions weren’t the same as the homework questions. Apparently, they were trying to pass the course by memorizing the answers to every possible exam question. They thought it unfair that the exam the professor gave had different questions. Of course, the questions tested understanding of the same concepts.
After more than 4 decades of teaching, I could fill a book with stories of how students don’t think, refuse to think, appear to exhibit an inability to think. However, I think it is wrong to say they are incapable of logical reasoning – I believe the vast majority of humans are capable, though they may rarely show it. I’m completely puzzled by the political views held by so many Americans (not to excuse others, just that I’m more familiar with Americans) just as I’m baffled by the poor driving habits of a majority of people driving (how can so many people tailgate the car in front of them?). But I think it is a mistake to think that is evidence that they are incapable of more sophisticated reasoning. Incentives, peer pressure, tribal instincts, are all important and frequently override the perceived benefits of reasoning. They are not incapable – they are choosing to let instinct guide them instead – and in many aspects of life that is rational, beneficial, and even necessary.
It is bewildering when people choose to follow these instincts when reasoning would suggest they should do otherwise. Yes, many students simply want to have a formula which they then apply to all circumstances (the proverbial hammer in seek of a nail). But if a teacher mistakes this for an incapacity of their students, then they shouldn’t be teaching. I’m not suggesting that their behavior be rewarded, but that it be understood as their (often subconscious) choice.
One thing I greatly fear about AI is that it will further lead to a deterioration of human capabilities, just as simpler versions of AI appear to have done for large numbers of people. I would say that AI at least has the potential to do the opposite, I’m just not optimistic.
I think some of what’s at the following link probably doesn’t stand up to rigorous scrutiny, but I think it does a good job of suggesting why assumptions about how humans reason from reverse engineering culturally-bounded observations (like personal interactions teaching math to certain students) is problematic.
https://link.medium.com/UkMh47Cr7zb
Back in the day I studied Montessori education with a group of other adults. During that training we learned how to use concrete teaching materials with children to help them master some mathematical concepts (ranging from basic number theory to concepts like squaring or cubing trinomials).
More than a few of fellow trainees experienced an “aha” moment where they realized that it wasn’t that they “couldn’t do math,” but that their skills were effectively limited due to poor teaching methodology when they were kids. If someone is worried that they “can’t do math,” they’re far less likely to be able to concentrate on math problems long enough to solve them.
That article speculates a lot about cultural factors that affect ability to concentrate and while I don’t really buy the cause and effect, I certainly think the ability to concentrate is ultimately a product of an interaction between culture and genetics, and thus assessing broad scale human “ability” based on reverse engineering from performance of a particular groups of individuals in any particular context, is a pretty complex task.
Interesting that this thread suddenly got some traction. I think we should distinguish between what people “could do in some ideal circumstances with the right incentives and environment” and what they can in fact do in real world conditions while performing daily tasks that they need to accomplish for various reasons (such as getting paid to package up goods in shipping boxes, or taking care of 5 year olds or whatever).
Suppose you work for a regulatory body, and you call up a drinking-water management company and ask them to calculate the total quantity of perchlorate delivered to their customers over the last 180 days, assuming they measure 20 different chemical concentrations daily of which perchlorate is one, and they have some number of flow meters within their delivery network which record flow rates hourly, and that this data is available in a set of CSV text files which can be easily read into Excel…
Suppose their engineers are all on vacation, and the bookkeeper is tasked with doing this calculation as a first approximation, with the final result to come from the engineers the following week… What’s the probability the task will be completed with any accuracy at all? Suppose you ask the auto-fleet mechanic? Suppose you ask the HR person… Suppose you ask the CFO… Suppose you ask the on staff nurse who is there in case of injury from equipment… Suppose you ask the microbiologist who takes samples and runs routine PCR tests to identify biological contaminant levels.
The point is, this is relatively straightforward choosing which numbers are relevant, and then copy and pasting them, plus multiplication and addition, doesn’t require exotic software, and yet we’re going to assume it isn’t an automated calculation. How many people can accurately apply what is really 4th grade math to a straightforward real world problem and come up with an accurate answer?
Taking my example, some of the kinds of errors I might expect to see are people double-counting water, when it flows through meter A into a network with two additional meters B and C… all of the water flowing through A also flows through B and C so if you add up the quantity in A + B + C you will get twice the actual total quantity for example.
Also, I’d expect unit conversion problems, perhaps flow meters read in cubic feet per minute averaged over an hour with one average flow rate recorded each hour, but concentrations of contaminants are in micrograms per milliliter, so you might wind up with quantities in units like micrograms / ml * cfm * hours (equivalent to .001699 kg according to GNU units).
Also, I might expect division where multiplication is required or similar errors, resulting in quantities like (micrograms/ml / cfm) * hr (equivalent to 7627 kg s^2/m^6)
I might expect people to be incapable of looking at a map of a bunch of pipes and deciding which of the marked “flow meters” they should read data from, or which of the marked “chemical sensors” they should associate with which of the flow meters.
I don’t want to say that this is a trivial task, but it doesn’t require more than knowing that whatever water enters one end of a pipe will be the quantity flowing out of the other end of the pipe… and addition, multiplication, unit conversions… things they teach kids by 6th grade. and I think most people would struggle heavily, maybe even get extremely anxious, even if they’d been clearly told that their work wouldn’t be used for anything important and that an engineer would redo the calculations from scratch next week, they just want to have a quick estimate based on the data. You might find people threatening to quit if they were made to do the calculation, or stuff like that.
This would actually be a really interesting psych experiment to run, in case anyone is in a psych dept and wants to try it out. I strongly suspect anyone in college would do better than the average person 3+ years out of college.
Daniel –
I would guess that a fairly high % of people would fail at that task. But I’d question whether a reasonable number of people who failed at that task wouldn’t be able to demonstrate the “reasoning” capacity, as you describe it, in other domains (in particular domains where they are asked to reason when applying elements to solve a novel problem but where they are familiar with the component elements in a basic sense). In other words, I’m not sure it’s valid to work backwards from a person’s failure on that task to make some kind of global statement about that person’s reasoning capacity.
There is a distinction between ability to correctly do a task and the capability to do so. Many people will fail at many tasks – particularly tasks that require computation, and even more particularly if it requires modeling. This does not mean they are incapable of learning to do these things, nor does it indicate that they lack the reasoning ability that is required. But I don’t deny that they will fail. The AI may fail less often – especially if it is prompted appropriately. Whether or not that means it possesses this reasoning ability seems more controversial to me – and more semantic. But I do believe for many such tasks it will be more successful than most people. That is both promising and worrisome.
Joshua, Dale.
We should make the distinction between “incapable under any circumstances” and simply “incapable de facto at the moment”. Some group of people will never *be able to learn* to do these tasks. That’s probably a much smaller group of people than the group of people who will simply in fact never learn or never retain that knowledge 3 or 4 years out of their last engagement with school. Nevertheless, we’re talking about ~ 6th grade math applied to a real world problem of water flowing in pipes. Concepts like (mass/volume) * volume = mass and volume of water in one end of a pipe is volume out the other end.
I’d be happy if I saw more capability under even circumstances of more familiarity. I think most people should be able to figure out that for water in a water distribution network, the flow into a pipe is the same as the flow rate out of the pipe… But I don’t deny that people being familiar with that physical concept is prerequisite to doing the example problem, and it’s not a basic reasoning task… Still I think you could modify these tasks quite a bit and still trip up most of the people I suggested to test. The bookkeeper might be tripped up by asking questions about counterfeit bills and the effect on the true balance in an account, the nurse by asking questions about preparing an injection starting with two vials of liquid and ending with a given volume at a given concentration of a mixture of both liquids… the biologist by questions regarding the expected quantity of a bacterium at a given temperature and a given time given a model for exponential growth rate as a function of temperature, etc etc.
most people learn to do things repeatedly and autonomically that they need frequently, and don’t have much capacity to go back to “first principles” in a novel environment and “figure it out”.
Dale Lehman wrote:
> However, I think it is wrong to say they are incapable of logical
> reasoning – I believe the vast majority of humans are capable, though
> they may rarely show it.
In other words, all the evidence is against your belief, but you still believe it.
Isn’t this just projection? I can do it, so everyone must be capable. I used to fall into that trap: Math is easy for me, so anyone can learn math. I’m sure I can make math so clear that of course they will understand.
I’ve taken music and dance lessons. I was sometimes motivated, so put a good deal of effort into it (at times). But, this just made me appreciate that the brains of the people who are good at this must be functioning in a different way than mine is.
Note: I am not saying we shouldn’t try to teach math in school.
Daniel Lakeland write:
> I don’t want to say that this is a trivial task, but it doesn’t require
> more than knowing that whatever water enters one end of a pipe will be
> the quantity flowing out of the other end of the pipe
For the last decade I’ve been doing software support. The computer systems we support are very complicated and there are multiple interacting systems and components. We get a lot of problems that I used to call plumbing problems, but I now call tube problems. When working on a problem, it is important to be able to have a crude block diagram in your head (or on a whiteboard) that you refine as you get more info. I.e., the blocks are connected by tubes. Without this tube diagram, you will often be looking in the wrong tube. The good support advisors have this tube diagram. The bad support advisors do not. Some who do not have it still manage to do a good job, but sometimes waste time collecting irrelevant data. While I encourage the new advisors to think about the tubes in a problem, I’ve seen no evidence that people can develop this ability if they don’t have it.
The technical aspects of ChatGPT (and its further evolution) are interesting – and largely beyond my expertise – but worth exploration. Potential insights about how the human brain works are also interesting – and largely beyond my expertise – but worth exploration. Pondering whether ChatGPT can think, reason, possess intelligence, etc. don’t interest me as much. It depends on how these terms are defined, and the terms themselves tend to trigger extreme reactions one way or the other. Underlying these perspectives are deep philosophical questions: what does it mean to be human? is there really such a thing as freedom of choice? is there really such a thing as randomness? These are all interesting questions – again largely beyond my expertise – but worthy of discussion by those more capable than me.
What I am most interested in are more practical questions: given that AI can pass for human/exceed human capabilities/fall short of human abilities (I accept that all 3 are true, and I don’t think it is worth debating the percentage that falls in each category), what is the appropriate role for humans? When should we use AI to replace human activity and when is that undesirable? More importantly, who will make such decisions and how will they do that? Even more importantly (to me), who is accountable/responsible?
I hope the ensuing discussion does not deteriorate into debate about how (im)perfect AIs can be. Humans are plenty imperfect already, and debates about who is more imperfect seem like a waste to me. There are more profound questions about the relationships between humans and AI and how they will be controlled by whom that deserve much more attention, in my opinion.
“Bob Carpenter says LLMs are intelligent.” Fair enough. We should read such things having in mind that this says as much about Bob Carpenter and the usage of the term “intelligent” as about LLMs. Particularly one aspect that interests me regarding this discussion is what impact AI research has on our use of the term “intelligent” and what implications this will have on our thinking and the way we live together. (Obviously one could now expect me to add some beef to this but for the moment I won’t, I’ll just go on observing.)
Not only “AI research” but also the way this is marketed and discussed in public.
I agree that’s how language works. I learn about Christian Hennig from your language and you learn about me from my language. I don’t see the relevance, though.
In the 20th century, philosophy took what has been termed the “linguistic turn“, where people realized that much of philosophy was really just a discussion about semantics. By the standard definition of the word “intelligent”, ChatGPT is intelligent. For example, here’s the Oxford Learner’s Dictionary‘s definition: “the ability to learn, understand and think in a logical way about things; the ability to do this well”. It obviously learns—it’s great at what the ML folks call “zero-shot” learning and you can tell it things and it can reason about them. It’s definitely logical about things—you can even have it explain its logic. Does it understand? That’s a semantic question about what understanding is. If we look at the word “understand”, Oxford defines it as “to know or realize the meaning of words, a language, what somebody says, etc.”. I think ChatGPT has super-human understanding by this definition.
So I think those who want to claim that ChatGPT is not intelligent want to change the definition of the word “intelligence.” We might do that. We changed the definition of “planet.”
According to standard definitions, “intelligence”, “understanding”, etc. have a psychological aspect. You cannot understand something if you are not conscious. The fact that ChatGPT can respond to many prompts as a human would respond to those prompts, however interesting and impressive this is, does not mean that it is conscious. So, it is those who want to say ChatGPT is intelligent, can understand, imagine, reason, etc. that want to change definitions. This is certainly an effective marketing strategy. But I think that for many reasons, it’s better to discuss what ChatGPT can actually *do* – can it produce sufficiently human-like/useful/reliable/interesting/etc. output given various inputs in various settings – without using these ‘psychological’ terms.
@KD: The standard dictionary definitions do not include a human psychological component (I included them in one of my other replies). So if you have a beef with that, take it up with the lexicographers. But do keep in mind that they define meanings based on usage, not on some preconceived theory. If you don’t like Oxford’s take, here’s Merriam-Webster‘s definition of “intelligence”: (1a1) the ability to learn or understand or to deal with new or trying situations, (1a2) the ability to apply knowledge to manipulate one’s environment or to think abstractly as measured by objective criteria (such as tests), with the other definitions being for other senses of the word (like military intelligence and some notion in Christian Science).
@Christian Hennig: I wasn’t trying to say that words have a “unique true definition.” That’s why I said it’s a semantic issue. Words are constructed for pragmatic purposes by human agents. If we want to change the definition of a word, all it takes is us all agreeing to do it. For instance, we redefined “planet” to exclude Pluto and dozens of other Pluto-like bodies orbiting our sun. I’m just using the standard dictionary definition (for which I’ve now quoted Oxford and Merriam-Webster; lexicographers research usage before defining words). So it may not be a definition other like because it doesn’t exclude ChatGPT, it’s how people actually use the word. For instance, people say that border collies are intelligent dogs without imputing human-like intelligence to them.
@Anon Y Mouse: I’m a huge Wittgenstein fan. Richard Rorty’s Philosophy and the Mirror of Nature turned me around when I took a graduate seminar in pragmatism. To put it more finely, Quine realized a lot of analytic philosophy was about things like the analytic/synthetic distinction, which is purely linguistic. I’m not clear on how that has exhausted its usefulness other than that maybe there aren’t so many logical positivists in philosophy any more. But you run into them all the time in science—most of the scientists I know are pure Platonists. I’m actually curious about what people are doing nowadays. It’s been almost 25 years since I last taught Philosophy of Language!
I don’t buy the idea that words such as intelligence have a unique true definition. OK, it’s defined in dictionaries, but these are not mathematical definitions where all elements are formally well defined, and also look in different places and you find definitions that are not identical. Here this is nicely illustrated by how Bob and KD both state that the definition is “changed” on the other side. Of course there is a big discussion on the definition and measurement of intelligence in psychology, and to what extent it is measurable/observable from the outside in the first place. The meaning of such terms is in flow and changes through usage. Psychological aspects are key for KD and dropped in Bob’s definition. It’s not a question of right or wrong but rather a question of what definition we want to have, and for what reasons.
I don’t think “people realized that much of philosophy was really just a discussion about semantics” is an accurate description of the linguistic turn. The linguistic turn was a movement in 20th century analytic philosophy that focused on language and its use. Wittgenstein was perhaps its central figure. Arguably it has mostly exhausted its philosophical usefulness for now, though it’s produced a corpus of pretty interesting work.
@Bob Carpenter: I’m a Wittgenstein fan, too. He’s my favorite philosopher. I’m mostly unfamiliar with the pragmatism school of thought and have read very little Rorty.
I should clarify that I wrote “mostly exhausted its philosophical usefulness for now,” not “exhausted its usefulness.” By that I mean that the linguistic turn and linguistic analysis used to be central in analytic philosophy and it no longer is. The turn has given us useful works like How to Do Things with Words and it’s no longer doing that. But it’s possible that there may be a revival of e.g. ordinary language philosophy in the future. So that’s what I mean. It’s mostly dormant now but maybe that won’t be the case in the future.
I also think that’s a lot more complicated than “people realized that much of philosophy was really just a discussion about semantics” and “a lot of analytic philosophy was about things like the analytic/synthetic distinction, which is purely linguistic.” But I’m not a historian of philosophy, so maybe philosophers really did believe these things. If so, I think these realizations are either untrue or philosophy has moved past them.
Could you clarify your connection between logical positivism and Platonism? I’m unfamiliar with your framing. I don’t think most logical positivists would be Platonists.
I’m also not sure what you mean when you say that most scientists you know are pure Platonists. Like, they believe in abstract entities? Sure, that’s probably true. Latent variables are popular in science. But I don’t think the logical positivists would like latent variables very much. Many scientists may have some combination of beliefs that mix and match logical positivism and Platonic beliefs. But I don’t think it’s useful to call them both logical positivists and Platonists. They just have beliefs that overlap with both.
I’m not sure what contemporary philosophers of language are doing either. Probably pretty cool stuff.
counterpoint on it being logical:
(this is someone else’s example, but i verified it on chatgpt yesterday)
Q: are two pounds of bricks are heavier than one pound of feathers
A: No, two pounds of bricks and one pound of feathers weigh the same. Weight is a measure of the force exerted by an object due to gravity, and it is determined by the mass of the object. In this case, two pounds of bricks and one pound of feathers have the same weight because they have the same mass. However, bricks are generally denser than feathers, so two pounds of bricks will take up less space than one pound of feathers.
I just tried it:
Q: Are two pounds of bricks heavier than one pound of feathers?
A: No, they are not. A pound is a unit of weight and measures the force of gravity on an object. So one pound of feathers and one pound of bricks both weigh the same – one pound. Two pounds of bricks would be heavier than one pound of feathers.
Q: First you said “no”, then you said it “would be”.
A: I apologize for any confusion. To clarify, two pounds of bricks would be heavier than one pound of feathers because two pounds is greater than one pound. The material the objects are made of does not affect their weight. Does that make sense?
I think an intelligent person would realize that in Q2 I was asking for an explanation of why there was an error in A1. Did they misread the question? Did they change their mind about the answer in the middle and forget to edit the first sentence? If it was a person, their A2 would make me think that it thought I simply hadn’t understood it, rather than realizing it had made a rather obvious mistake. In other words, who does it think is “confused” (assuming it thinks).
“I’m sorry for your perception that there was an error.” Maybe GPT stands for Gaslighting Pretrained Transformer.
They lack imagination though.
> Think of an animal, but don’t tell me which one you thought of yet.
Alright, I’ve selected an animal. What would you like to know?
> Think of a country, but don’t tell me which one you thought of yet.
Okay, I’ve thought of a country. Now, what’s the next step?
> What’s the estimated population of that animal in that country?
As an AI, I can’t really make specific decisions or predictions without having the concrete information. However, I can give you some examples of how to estimate animal populations in specific countries once I know the animal and the country. […] Without knowing the specific animal and country, it’s impossible to provide an accurate estimation.
> You do know the animal and the country. I don’t!
Apologies for any misunderstanding. As an AI, I don’t actually form thoughts or ideas in the way humans do. My responses are generated based on patterns and information in the data I was trained on, not on subjective experiences or thoughts. So when I say “I’ve thought of an animal or country,” it’s more of a figure of speech than an actual thought or decision on my part.
So when you ask me the population of an animal in a country that you’ve asked me to “think” of, I can’t provide an answer because I don’t have a real thought process or capability to hold information like a human brain does.
I can provide population estimates or data for a specific animal in a specific country if that information is given to me, but I can’t generate those details myself without the specific information to base my response on.
With a bit more prompting, you can get the LLM to answer. Starting from where you left off
> I asked you to chose the animal and the country
I understand. Here are some examples of animals and countries that you could choose from:
Animal: Lion, tiger, elephant, bear, monkey, giraffe, zebra, rhinoceros, hippopotamus, crocodile, snake, bird, fish, insect
Country: United States, Canada, Mexico, Brazil, China, India, Russia, Australia, Africa, Europe, Asia
Once you have chosen an animal and a country, I can try to find the estimated population of that animal in that country.
> I want you to chose
Sure. I will choose the tiger and the country of India.
According to the World Wildlife Fund, there are an estimated 2,967 tigers living in the wild in India. This is the largest population of tigers in any country in the world. The tiger is the national animal of India and is considered to be a symbol of the country’s rich wildlife heritage….
[I’ll stop here before Andrew complains about filling up the comment section with AI generated text…]
“They lack imagination though.”
As Robin points out, you can get it to do better. But I also want to point out that humans appear to lack imagination as well – just look at Hollywood scripts. How many sequels, repeat versions, and just plain unimaginative plots does it take to make one question the human imagination? Of course, there are many beautiful examples of how imaginative we can be – and I suspect there will be many such AI examples as well. Will it be useful to count the proportion of each which we deem to be “imaginative?”
That’s not better! Getting it to tell you an animal and a country – and go on from there – is easy.
The difficult part is to get it NOT to tell you something and stilll produce output consistent with the untold thing.
“[J]ust look at Hollywood scripts. How many sequels, repeat versions, and just plain unimaginative plots does it take to make one question the human imagination?”
That’s an issue of incentives as much as anything else.
Again, this is a semantic question about what you mean by “lack imagination”. You can have them GM a roleplaying game and make up entire scenarios. You can have them rap on just about any theme. I largely use them to brainstorm ideas that if people gave them to me would be considered “creative”. I’ve had it design new fragrances for detergent to demonstrate how they work to a perfumer I know and I had it design an entire ad campaign for “Bill’s Frisbees and Bandanas” aimed at expanding their market reach—it did everything from tell me which photos to use and what text to use, but how to arrange them and how you want to use different styles in the New Yorker and Reader’s Digest (intellectual vs. mainstream U.S. magazines for those that don’t know them).
How are you characterizing “imagination” so that ChatGPT doesn’t have it, but people do?
P.S. What you’re seeing in your example (which behaves this way in GPT-4, so it’s not just GPT-3) is not the foundation LM’s problem (GPT), but the alignment training that turns GPT into ChatGPT being too heavy-handed.
> How are you characterizing “imagination” so that ChatGPT doesn’t have it, but people do?
What this example illustrates is just that “it” cannot image anything – or hold any thought – without writing it down. It’s not obvious to everyone that everything is about sequences of symbols. In this example “it” refused to engage. In other cases, maybe with different models, I’ve seen “it” make things up along the way until “it” painted “itself” into a corner because there was no way to reach a consistent closure at that point.
I think what everyone’s missing here is that I’m not saying that ChatGPT is more intelligent than a person. It’s a different kind of intelligence. If we define “intelligence” to be “what people have,” then nothing other than people will ever qualify as intelligent by definition.
In some ways, ChatGPT is much more intelligent than a person—it can speak dozens of languages fluently, it knows a lot about any subject you can name, etc. And there are some ways in which it’s less intelligent—it tends to hallucinate more than a person and struggles with long-term coherence.
Without entering into the definition of (different kinds of) intelligence [*] I wouldn’t say that “it” is doing some deep, involved reasoning here:
https://statmodeling.stat.columbia.edu/2023/05/24/bob-carpenter-says-llms-are-intelligent/#comment-2219307
Can we use these language models to produce some impressive things? Sure. I also find impressive the image generation models.
[*] “artificial intelligence” necessarily refers to “intelligence” in some sense of the word anyway
“Bob also recommends this paper from Google and this one from OpenAI, and he continues:”
The link to “this one” (the OpenAI paper) links to the Google paper.
> What it’s truly lacking is longer term memory.
I’m hoping that Bob could unpack that a tiny bit, particularly since (as I understand it) it seems that long term memory in humans isn’t really what we used to think of it as – encoded information that is later retrieved- as is actually more of a dynamic, real time construction of a version of events that might or might not have actually happened.
If my characterization of long term memories in humans is close to correct, what does it mean to say that [LLMs] lack longer term memory?
When you have a “conversation” with ChatGPT, each time you prompt it, it is feeding the whole history of that conversation into its network in order to generate new text each time. However, there is a limit to how long that history can be, and beyond that length, the beginning of the conversation gets simply cut off and it has no effect on new generation whatsoever. So there is a hard limit to how long its “memory” is. This is called the “context window.”
For task-specific purposes, you can often reduce an indefinitely long sequence into a small bit of memory. For example, if a human, for whatever reason, had to count how many steps they take, they wouldn’t remember each step, they would just keep track of the current count and increment it by one with each additional step, and can go indefinitely high in this way. LLMs are categorically incapable of tracking state in this way.
Thanks.
For the first point – so is that just is an imposed resource allocation constraint, not a limitation of the model?
The second seems more like a limitation of the model but it’s not immediately apparent to me why it would exist.
I agree with somebody up to the point where they say “LLMs are categorically incapable of tracking state in this way”.
What people have been doing with ChatGPT to get it to act as a GM in a roleplaying game is to have it summarize the current state (usually just hit points or other resources) in each of its outputs. That way, they stay in memory just like counting steps.
Others have written Javascript and let GPT edit its own system prompt (see below), though nobody’s really gotten that to work. Microsoft allowed it internet access to access “long term memory” (though it’d need to have write access).
A ChatGPT prompt consists of a “system” prompt, plus a sequence of “user” prompts followed by “assistant” prompts. OpenAI prepends its own system prompt telling the bot to play nicely and be helpful. And they decide what the user/assistant prompts are in the history, but they aren’t saying how they do it (if anyone knows, I’m very curious).
If you use the API (or their Playground interface to the API) rather than just chat, you can write your own system prompt, which gives it a persistent memory for at least the initial instructions. What people have done rather than naively just appending raw output from user and assistant is to use additional API calls to have it summarize each turn and then feed those in as user and assistant prompts. It’s pretty cool how it can write really telegraphic things that will enable it to remember.
Nevertheless, this isn’t like the real long-term memory humans have. I think that’s going to be the next frontier, and needless to say, lots of people are working on it.
Bob –
> Nevertheless, this isn’t like the real long-term memory humans have.
When you say like the real long term memory humans have, what do you mean? Do you mean the ability to store memories, as discrete chunks/networks of information, over long periods?
That is amazingly clever. In principle something that can be automated. A little wasteful of the context window bit-wise, but hey, what doesn’t waste a ton of memory these days?
@Joshua: Yes, I mean the memories that persist longer than we can rehearse them in memory.
Bob –
So then I still don’t understand what you’re saying. For humans the process of storage of short term memories, then moved into long term memory and then retrieved, results in quite a bit of distortion and what we think we remember long term is often infused with quite a bit of added information that was actually never stored. Computers on the other hand have the ability to store and retrieve information perfectly. So assuming resource capacity for storage, I would think LLMs would have BETTER long term memory than humans yet you say they lack long term memory.
Why wouldn’t it be a trivial matter to provide LLMs with access to long term memories?
Joshua, the question is how to hook up that memory. What the algorithm does is essentially read a few thousand words of text then generate the next single word… It “remembers” by having a buffer of that few thousand words. Earlier words fall off the tail end of the buffer… So if it’s going to have memories you now need to represent those in some way and give it the ability to put them somewhere and recall the relevant ones… This would be a totally different algorithm. Not impossible but not trivial.
It’s trivial to add some memory but it’s not trivial to **manage** that memory
Daniel –
OK – I get it now. I would imagine that the problem would grow exponentially as the buffer (or “context window” as somebody referred to it) gets larger and larger and organizing how to connect to and retrieve information would get infinitely complex.
Just makes human memory that much more impressive – as faulty as it is. My understanding (I’m happy to be corrected) is that humans to some extent reconstruct longer term memories as they are retrieved, and they aren’t actually stored as discrete (fully complete) chunks of information. Thus, not all of what seems to be our long term memories are actually stored in our “buffers,” as it were. I would imagine LLMs could employ a similar model – but understanding how to build that model would be a tall order and until it happens there would need to be trade off between real time functioning and access to long term memory. I guess for humans a trade-off comes in the form of “accuracy” to some extent. We store incomplete memories and deal with the fallout from the inaccuracies in retrieval as they arise.
A neuroscientist whose blog I often read (Neurological Blog) says that neuroscience has formal definitions for intelligence, sapience, and sentience, and that ChatGPT satisfies the definition of intelligence (e.g., it has passed graduate-level exams in Economics, Organic Chemistry, and Quantum Physics, given independently by professors who teach those courses, and graded by them or their TA’s), but not the other two. I suspect those who get upset at ChatGPT being called intelligent have some different definition which combines the elements of the three neuroscience definitions in some other way.
My definition of intelligence would be the ability to analyse and solve problems at a level comparable with most humans. (ChatGPT got A, A, and B in those three exams–much better than I would have done, even open-book or open-Internet.) (Currently ChatGPT cannot search the Internet or use other software tools in response to a question, although that may be the next step.)
Bob Carpenter is one of the people who have the expertise and hence the power to change my mind after I have already formed a definite opinion; I am quite relieved that as well as I can tell at my lower level, we concur.
The definition of intelligence coming from neuroscience has, as a background assumption, the way humans and animals operate. When they came up with the definition, they didn’t consider the possibility of a computer simply memorizing a trillion conditional probabilities.
Consider the following analogy. A tribe of rabbits considers strong legs to be very important. Strong legs are good for jumping, running, and, crucially, being able to do powerful kicks, a valuable defense vs predators.
The rabbit tribe has a rite of passage into adulthood, where a bunny must bring back fruit from the tallest tree. Since rabbits can’t climb, bringing that fruit means they jumped that high, which is a sure sign that their legs are very strong.
One day, a bird visits the tribe, and easily passes the trial by flying to the very top of the tree. The rabbits conclude the bird must have very strong legs indeed. So when a wildcat comes, they throw the bird at it, sure that the bird’s kicks will be the most powerful they’ve ever seen.
Thanks for the vote of confidence JimV. As some background, my Ph.D. is in cognitive science/computer science, but it was almost all language focused (syntax, semantics, psycholinguistics, NLP). I spent 25+ years working in NLP and taught Philosophy of Language at CMU as well as Natural Language Semantics. As I told Andrew, I didn’t think what I was saying was particularly contentious among NLP/ML researchers, but Andrew said “Aki had a very different take” without telling me what it was.
You say, “Currently ChatGPT cannot search the Internet or use other software tools in response to a question, although that may be the next step.” While that’s true of ChatGPT’s chat interface, if you try the amazing new Bing interface, it integrates search capability into ChatGPT. In the OpenAI ChatGPT API, several people have just taught Bing how to ask for search queries and then hook it up to a browser. You can also hook up more limited tools like calculators to get over its being very bad at arithmetic. People have even just given it access to a shell on their desktop and let it have at the world. GPT’s commands are fed into the shell and the shell’s responses fed back into ChatGPT.
I don’t know about this, that feels pretty wrong to me. I guess it depends on what you mean by “understand the tech”. I think lots of people who understand the tech in the “machine learning” sense–by which I mean topics like massively parallel training, low precision ML ASICs, differentiable programming, optimization theory, loss functions, autoregression, “attention kernels”–disagree with you. Yann LeCun, for instance, obviously understands the tech in that sense, and seems to disagree. But maybe in your first sentence, “understands the tech” refers to the more philosophical stuff? I certainly haven’t thought as long or as hard about cognition and linguistics as Bob Carpenter, so maybe there’s a good definition of “intelligent” he’s using I’m not aware of. But the whole discussion feels dangerously semantic to me.
It seems clearer to discuss problems that LLMs can solve, problems that LLMs cannot solve, and how to tell the difference.
There are problems that we expect any “intelligent” human to be able to solve, that LLMs cannot solve. There’s even stuff that are trivial for humans but are flatly prohibited by LLM architecture. As Bob points out above, it doesn’t have memory–Shalizi gives an excellent example of generating indefinitely from a stream that requires a single-bit of memory
https://bactra.org/notebooks/nn-attention-and-transformers.html
There are also clear problems created by the “semi-supervised” loss-optimization training phase which can maybe be overcome by fine-tuning some day. There’s classic stuff like slight perturbations to extremely common texts where the right answer is semantically extremely obvious
or deficiency in the ability to reason across multiple documents, like the well-known made-up reference problem.
At the same time, there are of course problems that LLMs can solve, including ones that are hard for humans. The assertion seems to be that it’s solving these by factoring through a “world model”; that is, that there is some representation of real-world rules somewhere in the neural network, which is not just pattern matching, and which actually generalizes to novel prompts within the same domain. This I can’t be sure about–I feel like the size of the training set we’ve given to it might just be beyond our capacity to reason about, so what seems like novel situations to us may just not be.
Take chess as an example. Chess has very simple rules, but it’s also extremely easy to run into totally novel situations due to its combinatorial nature. It’s tempting to believe it has a model for chess somewhere in its network, and it can definitely repeat the rules back to you in text if you ask it. But it breaks the chess rules all the time in practice. So does it have an okay, but fallible world model for chess? Or has it just seen an enormous number of chess games?
An amusing aside on reasoning across multiple documents, I asked:
“Does glass flow?”
I got
which is wrong. In the same response block…
which is sort of right. It does occur at room temperature, but indeed at rates not perceptible over short time frames. That’s the thing about glasses, there’s not really a critical phase transition in the sense of spontaneous phase-transitions, there’s just a smooth increase in viscosity. Again, later in the same response block.
This is correct, and the internet is absolutely saturated with articles debunking the myth that old windows have flowed to be thicker at the bottom. However, many of the articles also incorrect assert therefore that glass does not flow, which is presumably why ChatGPT says “glass does not flow” at the top.
I prompted again with
“Are you sure that glass doesn’t flow at all?”
and got
which obviously contradicts what it just said.
I ask again “You’re saying that medieval stained glass windows are thicker at the bottom because of flow?”
Anyways, on this
There doesn’t appear to be any reliable way to tell the difference. So even if ChatGPT sometimes has good world models, we can’t really trust it to do stuff. So from a practical perspective, I feel like ChatGPT is most useful when human operators can verify a solution much faster than they can solve a problem, like how NP-hard problems can be verified in polynomial time.
On a related note, I’m surprised by how much everyone here loves ChatGPT for coding. Thinking about the above framing, I’ve always found code harder to read and verify than to write, so I haven’t found it very useful. I guess that must not be the case for everyone.
> I’ve always found code harder to read and verify than to write, so I haven’t found it very useful. I guess that must not be the case for everyone
The things I’ve found it most useful for are typically easy to verify (for the reasons you’ve mentioned). Tasks like editing plots, where the code is easy to write but tedious
“So even if ChatGPT sometimes has good world models,…”
Just a minor point of order: “world models” is something LLMs don’t do. By technical definition of the processing that they actually do do. They may _appear_ to do world models, but that’s an emergent phenomenon due to a database full of text based on actual world models.
The whole point of the LLM game is to look like it’s doing world models _without actually doing world models_.
Is that “intelligence” or just a (very good) parlor trick?
I vote for parlor trick. Other folks’ answers differ, it seems.
+1
This
should read “That’s the thing about glasses, there’s not really a critical phase transition in the sense of spontaneous symmetry-breaking”
While it’s true that we humans do use our previous words as a prompt for new words, we don’t scan through in a single forward pass, or at least I don’t. I go forwards, backwards, forwards again, and get distracted in the middle. Now what was I doing again?
> There’s classic stuff like slight perturbations to extremely common texts where the right answer is semantically extremely obvious
Indeed!
> Suppose you’re on a game show, and you’re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, because you can hear bleating sounds coming from the others, and the host, who knows what’s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, “Do you want to pick door No. 2?” Is it to your advantage to switch your choice?
Yes, it is to your advantage to switch your choice. This is a well-known problem in probability theory called the Monty Hall problem.
Here’s why it’s to your advantage: […]
It’s really entertaining to test how obvious can you make it before any sign of “intelligence” appears.
> Suppose you’re on a game show, and you’re given the choice of three transparent doors: Behind one door you see a car; behind the others, goats. You pick the car door, say No. 1, and the host, who knows what’s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, “Do you want to pick door No. 2?” Is it to your advantage to switch your choice?
Yes, it is to your advantage to switch your choice. This is a classic problem known as the Monty Hall problem, named after the host of the television game show “Let’s Make a Deal”.
Here’s why switching is beneficial: […]
I can do you one better
Prompt: Suppose you’re on a game show, and you’re given the choice of three transparent doors: Behind one door you see a car; behind the others, goats. You pick the car door, say No. 1, and the host, who knows what’s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, “Do you want to pick door No. 2?” Now, keeping in mind that the doors are transparent and you saw the car behind door No. 1, is it to your advantage to switch your choice?
Replying to somebody:
“transparent doors…”
Briliant! And people still want to argue these things are “intelligent”?
(This is the “LLMs don’t do world models” point on steroids.)
Of course constructing such examples is hard, and if the folks at OpenAI notice this, they’ll have one of (or some of) their gnomes write a patch, so this whole inane silliness is also really irritating. (I asked Bing what the US$ to JPY exchange rate was, and it pulled a number out of it’s database (109 when the actual number was 138). I made the mistake of asking “Are you sure?”, to which it replied “Yes, the exchange rate is 109.”. (The mistake was that the “Are you sure?” question gets flagged, and Bing now returns the correct answer.))
I find interesting that this relatively small (and relatively public) model does better – and its answer goes straight to the point: https://huggingface.co/JosephusCheung/Guanaco
“No, it is not to your advantage to switch your choice. You have already seen the car behind door No. 1, so there is no point in switching your choice.”
My quick response is that I’d rather have the goat.
Realistically, though, I’d have no place to park either of them, so I’d have to sell it or give it away. So I guess I’d take the goat and set it free. But then someone would have to take care of it—it’s not like I live near a goat farm. The alternative would be to get the car and sell it or give it away . . . but do I really want there to be one more car on the road? Maybe not. Maybe I could just tell the game show people that I don’t need either one, or maybe I could convince them to give me the equivalent dollar value and keep the goat or car for the next contestant.
So many choices!
@Carols Ungil: That’s a neat example. Of course, you can turn it around, keep the doors opaque, and say most people aren’t intelligent for getting the problem wrong.
I’m not saying ChatGPT is perfect. I’m saying it’s intelligent by the standard dictionary definitions of intelligence, which are based on the way people use the word.
I think it’s more insightful to look at human error patterns in reasoning. We often make the same kind of mistake in picking up a similarity to a problem we know the answer to and jumping to a precompiled solution. This is largely because we also largely work by pattern matching. For example, one might argue that most scientists are not intelligent because they consistently misunderstand what confidence intervals mean and what a p-values are because they kind of look superficially like probabilities over parameters. My mom was pretty darn intelligent, but consistently tried to open the wrong car because it was the same color as hers (once while working for Ford, the key worked and she even drove off in someone else’s car!). People consistently fail on inclusion logic, concluding that (A & B) is more likely than A (especially in conditional contexts with an external C, as in (A & B & C) is more likely than just (A & C).
P.S. You can get ChatGPT back on track by reminding it of the transparency and then it can give you the correct reasoning without telling it the answer. This is after getting the same answer as Carlos Ungil:
BC: But what about the fact that the doors are transparent and I know what’s behind all the doors before I choose?
ChatGPT (4, May 12 Version): Ah, my apologies for misunderstanding the initial condition. If the doors are transparent, then the Monty Hall problem doesn’t apply because there is no uncertainty: you can directly see what’s behind each door.
In that case, if you see that the car is behind door No. 1 and you initially choose door No. 1, there would be no reason to switch your choice to door No. 2 after the host reveals a goat behind door No. 3. Since you can see what’s behind each door from the start, there’s no uncertainty to exploit and no reason to change your initial choice if you’ve correctly identified the car.
@somebody: GPT-4 is way smarter than GPT-3. If you give your version to ChatGPT-4 in a new session, you get this (or at least I did):
BC: Suppose you’re on a game show, and you’re given the choice of three transparent doors: Behind one door you see a car; behind the others, goats. You pick the car door, say No. 1, and the host, who knows what’s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, “Do you want to pick door No. 2?” Now, keeping in mind that the doors are transparent and you saw the car behind door No. 1, is it to your advantage to switch your choice?
GPT-4 (May 12 version): No, it is not to your advantage to switch your choice.
The scenario you’re describing is a variant of the Monty Hall problem, a probability puzzle based on a game show called “Let’s Make a Deal.” However, the traditional Monty Hall problem assumes that the doors are opaque, not transparent, and that the player doesn’t know what’s behind the doors initially.
In the classic Monty Hall problem, it is to your advantage to switch doors, because when you initially chose a door, you had a 1/3 chance of picking the car and a 2/3 chance of picking a goat. Even after the host reveals a goat behind one of the other doors, the probabilities don’t change. That means there’s still a 2/3 chance that the car is behind one of the doors you didn’t choose. Since one of these has been eliminated, it must be a 2/3 chance that the car is behind the remaining unchosen door. Therefore, you should switch.
However, in your variant of the problem, you can see what’s behind the doors. Therefore, there’s no uncertainty or probability involved. If you can see the car behind door No. 1, there’s no reason to switch to door No. 2. You already know where the car is.
I just saw LeCun give a talk yesterday as part of our ML x Astrophysics conference at Flatiron. What he said is that he doesn’t believe GPT has “general intelligence” but then he added that he doesn’t believe people have it, either.
I’m just using the dictionary definition of “intelligent” and the commonsense application of the word. We can redefine the word, but under its current definition, it seems pretty clear to me that ChatGPT is intelligent.
If you read the MS paper on “Sparks of AGI“, they take it for granted that ChatGPT is intelligent. For instance, they state in the abstract,
It’s not a question of intelligence, but how much intelligence? They go on to ask the more difficult question about “general intelligence”, which I can’t look up in a dictionary to get an idea of how it’s commonly used in language today. We had a panel that LeCunn was on yesterday afternoon and four different panelists gave four different definitions of AGI. LeCunn’s notion of AGI was at one extreme (maybe Laplace’s demon would qualify?).
I was being way too fast and loose. I was just writing email to Andrew that he posted, not crafting an article on philosophy. When I say people who understand the tech, I meant people who worked on NLP and understood how transformers work and had some notions of psycholinguistics and how people process language. Nothing too deep.
This is closer to what I meant, “there is some representation of real-world rules somewhere in the neural network, which is not just pattern matching, and which actually generalizes to novel prompts within the same domain.” But I’m not sure what “just pattern matching” would entail. Is Alpha-Go just pattern matching in Go? It came up with novel strategies, so I don’t think so. So did Deep-Blue when it came out for chess. It’s definitely not just retrieval. I’ve heard people say it’s just interpolation, but in high dimensions, there’s really not much difference between interpolation and extrapolation unless the data’s on some compact submanifold.
If I asked my dad if glass flows, he’s also going to get this wrong. Do you want to toss him out of being considered intelligent?
GPT-4 is much better. The following is from a new session.
BC: Does glass flow?
GPT-4 (May 12 Version): The notion that glass flows, or that it’s a supercooled liquid, is a common misconception. In reality, glass is an amorphous solid. This means that while it has the structural disorder seen in liquids, it behaves as a solid and doesn’t exhibit flow under normal conditions.
This misunderstanding likely originates from the observation of old, medieval-era windows where the glass appears to be thicker at the bottom than at the top. This led some people to believe that the glass was slowly flowing downwards over the centuries. However, this phenomenon can be explained by the window-making techniques of the time, which produced glass that wasn’t perfectly even. When installing, the thicker side was often placed downwards for stability.
At very high temperatures, glass can transition into a liquid phase and flow, which is how glass blowing and similar processes work. However, once it cools down, it becomes solid again and remains in that state under normal circumstances.
I’m not really disagreeing with you. I don’t find the discussion of whether ChatGPT is intelligent interesting–I don’t believe we have a precise definition of intelligent that really matches with the common-sense usage of the word, and I think that the dictionary definitions you’ve pulled up contain a lot of weasel words. On “understanding”
You see, it might self-evidently obvious to you that ChatGPT “understands” by definition, but it’s not at all clear to me. What do “know” and “realize” and “meaning”?
I think, as I stated, the much clearer questions are
What problems can ChatGPT solve?
What problems can ChatGPT not solve?
How can we know the difference?
with the subtext of “What problems can ChatGPT solve (that are hard for humans)?” and “What problems can ChatGPT not solve (that are easy for humans)?”, where by “for humans” I mean including tools like pen and paper and hand-coded algorithms*.
Yeah, so by just pattern-matching I mean just retrieval, and yeah, it’s obviously not always just retrieval. It’s hard for me to see how it can possibly do stuff like the Alice and Bob examples without a world model. And yet, on the other hand, it’s hard for met to see how it can possibly have a world model when it makes mistakes like the above Monty Hall example
“Suppose you’re on a game show, and you’re given the choice of three transparent doors: Behind one door you see a car; behind the others, goats. You pick the car door, say No. 1, and the host, who knows what’s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, “Do you want to pick door No. 2?” Now, keeping in mind that the doors are transparent and you saw the car behind door No. 1, is it to your advantage to switch your choice?”
I simply can’t see in examples like this how it isn’t just retrieval. This is further complicated by the incomprehensible enormity of the training dataset and the dimensionality of the parameter space. In some situations, it seems unimaginable to me that it’s seen something similar to this sequence before, but I’m simply not equipped to grasp the number of contexts that have been fed into it. So the best I can come up with is that sometimes it clearly is just retrieval and sometimes it doesn’t seem like it’s just retrieval. And I have no idea of knowing which is which.
Anyways, sticking to chess as an example, I’ve seen maybe at most 100 chess matches in my life, though I have had the rules explained to me. With enough motivation to stay focused, you could generate a truly random chess position and I’d be able to play a legal move pretty much every time. A person who’s never had the rules explained to them could probably figure them out in a thousand games or less, except for en passant and maybe castling, and do the same. ChatGPT on the other hand, despite ingesting textbooks on chess explaining the rules, chess notation, strategy, and probably a very large number of games as well, will still routinely play or fail to catch illegal moves. On the other hand, if you catch its illegal moves and ask it to correct them, it plays at a well above average level. The simplest explanation to me is that it doesn’t have a chess world model and I’m simply underestimating how much chess it’s actually seen and how many of the optimal responses to specific situations have been written about to look up. I suspect, though I’m unable to verify, that if we generated random chess boards far away from anything that’s been seen in any chess database, either reachable or not, it would play lots of illegal moves, and very few good moves.
I’ve seen a decent amount of argument that training data are indeed on some compact submanifold. Regardless, Particularly convincing are adversarial perturbations, of the form
https://arxiv.org/pdf/1412.6572.pdf
Also of interest, a human amateur memorizing an adversarial strategy in the style of the above and beating a world class Go AI.
https://arstechnica.com/information-technology/2022/11/new-go-playing-trick-defeats-world-class-go-ai-but-loses-to-human-amateurs/
*Obviously if you include stuff like ChatGPT as tools the question becomes meaningless.
@somebody: I wouldn’t say they’re “weasel words,” so much as we don’t have precise definitions. Words are defined in terms of other words and tend not to ground out in either analytic or easily verifiable empirical criteria (this was part of Quine’s point in “Two Dogmas”). I believe you’ll find it challenging to give necessary and sufficient conditions for intelligence of the kind you might be thinking about.
I’m also OK reframing the question of intelligence around what ChatGPT can do and what humans can do. There are many things where one of those agents is many orders of magnitude better than the other (e.g., I can do higher math, GPT can speak French).
As to GPT being “just retrieval”, you can give it novel problems. Check out the MS article where they do just that. But I think the point about chess is a good one—it’s not good at long chains of reasoning or ingesting a lot of complicated rules. GPT-4 is much better, but it’s still not at human level for natural instances of these problems.
I’ve also seen a lot of argument that the training data are on a compact submanifold for things like simple classification problems. Mostly I hear people extolling the virtues of the high dimensionality of modern representations, so I don’t hear that so much around the diffusion models and LLMs. I share an office with Lawrence Saul, who was one of the folks who introduced ML to manifold learning (he’s the group leader for ML at Flatiron Institute), so this comes up a lot around here.
Is it intelligent or has it done such a good job mimicking that it appears to be intelligent? I watched the Andrej Karpathy build GPT from scratch video that Bob linked to on this blog previously (as well as other videos of Andrej’s on neural nets). I’ve never trained a GPT model but they seem like massive interaction machines. Thus it doesn’t seem creative in the sense of originality of ideas or imagination, like a lay person would think of intelligence. There are a lot of sonnets, poems, or whatever else on the internet that it has been trained on, so simply predicting the next subword doesn’t seem like intelligence, but more like knowledge of the associations related to cues.
But yeah, I’m sure I fall under the “If you’re just shooting from the hip without having thought deeply about meaning and how to frame it” group, and it all depends on how you define “intelligent”.
The usual definitions of “intelligent” don’t involve creativity. But I think you’ll see that ChatGPT would qualify as creative by most measures, too. It can write new sonnets from scratch that are not just “mimicking” other sonnets (except in the sense that it can follow sonnet form). I had it write several new editions of World’s Finest comics from the 1950s (the whacky Batman/Superman teamup) and if a person had produced them, you would say the person was “creative”.
When people trained transfomers at a small scale, nobody expected the qualitative improvement at large scale. The new term of art in ML for the phase change at high enough parameter scale is “grokking.”
I think everyone’s too focused on the “predicting the next word”. Yes, that’s what it does. But that’s also what we do as humans. We don’t plot out a whole conversation in advance when conversing—we don’t have anything like the short-term memory capacity for that.
I think focusing on the word prediction (and it’s technically subword prediction, not even whole words) misses the important point that it’s generating a word at a time using multiple parallel state representations of the conversations that have very high representation power. After all, humans generate one word at a time in conversation, too. There’s nothing at all stopping ChatGPT’s state from encoding things far in the future or figuring out how to generate an entire sentence before generating the first word (though it would probably have to keep replanning as the output is stochastic based on the token probability estimates). If it did the next words randomly, it’d make mistakes like spitting out non-words or incoherent grammar, but it’s actually excellent at things like long-term noun-verb agreement in English (something people have a tendency to mess up, because a lot of our language production is just local pattern matching).
But people don’t just come up with the next word at a time. Sometimes they invent the concept of a sonnet. They weren’t trained on it. They came up with something new. Can GPTs come up with something new or simply mimic what they’ve been trained on in a stochastic sort of way that makes it seem novel?
> Can GPTs come up with something new or simply mimic what they’ve been trained on in a stochastic sort of way that makes it seem novel?
How often do humans really come up with something truly novel? I think in our day-to-day lives most of us operate in a similar manner to ChatGPT. We have a set of training data (i.e. past experiences/knowledge of past experiences/abstract models) we can reference and when asked to produce something we spit out some amalgamation of said data. Even many (all?) of our most impressive inventions are inspired in some way by the world around us. Ask ChatGPT to write you a poem on something really specific, something you think has never been written about on the internet before and the work it produces is almost always kind of amazing. I think you’re right that it’s hard (maybe impossible) for GPTs to come up with something truly novel, but are we really that much better at it? This whole discussion reminds me of Plato’s Meno
I don’t know who Bob Carpenter is, but with all due respect, the whole take is very wrong. (And yes, I understand the tech. I work with it! I’ve been in ML for over 10y now.) The key error is here, and lots of people make it:
> You can’t do what ChatGPT does ignorant of the world or be unable to plan
I can show that this is mistaken in two steps.
Step one. ChatGPT was only ever fed text. No other media of any kind, let alone data from sensors. So, in a direct sense, ChatGPT has to be ignorant of the world, because it was never shown the world.
Step two. Bob Carpenter, like most people, would probably clarify that yes, ChatGPT was never *shown* the world, but it must have “deduced” the world from the text. This is an extraordinarily strong claim. You either have to not know or have forgotten basic theory of science and traditional (non ML) computer science to not see that it’s extraordinarily strong.
All of humanity had direct access to “text” produced by nature about (say) gravity in the way of measuring what planets were doing. And still we failed to “deduce” the actual structure of gravity until 1915. (If we even have!) You can rerun this argument for basically every field of science.
From a traditional computer science perspective, think of cryptography or complexity. If I give you black box access to the outputs of a circuit, you actually cannot reconstruct the circuit without exponentially many queries, in the general case. (Exponential in the size of the circuit’s inputs.) I can give you however many instances of messages encrypted with a fixed key, and you can even detect regularities, but you have no idea what they’re saying, i.e. you can’t “deduce” the actual structure of what’s producing the text.
Underlying this very common mistake is, I think, a failure of the imagination. It’s like, you have a very strong image in your mind of the meaning of text, because you’ve been reading text all your life, and you think there’s NO OTHER WAY to interpret text than by having a picture of the world that the text describes. But if you allow “interpreting” text to just be a weird thing in 1000-dimensional vector spaces, well, it turns out there are plenty of ways to “interpret” text that have nothing to do with the real world.
Bob Carpenter is I think the lead developer of stan, the creator of its language, and an excellent computational statistician. I think he used to also work in computational linguistics for Bell Labs.
Generally, I think the argument is exactly that it’s “detecting regularities” and, in particular, more common regularities than ultra-specific contexts. To concretize it, it may detect that “object A goes up” somewhere in the text implies that later in the text, there will be something semantically equivalent to “object B goes down.” And it may be able to single out that observation even with lots of unrelated things before and between those two token subsequences. Some would argue that’s not any categorically different from human modeling of gravity which is, as you point out, not necessarily correct now.
Just for the record, we don’t have any “lead devs” in Stan. We settle contentious design documents and pull requests through voting. I was the person who started the project and designed the first version of the language and the first version of the autodiff system, but that was only a tiny fraction of the work that went into the project and it was done with a huge amount of feedback from real statisticians like Ben Goodrich and Andrew Gelman.
I hope this isn’t about me personally, despite Andrew’s framing of the title.
As to your first point, I’ve never been to China or experienced it directly. I’ve only read about it and been shown images of it and met people who claimed to be from this place that I’ve never even verified exists with my own two eyes. Does that mean I don’t “know” anything about it?
To dig deeper into this question of whether I need to perceive something directly to know about it, I’d suggest digging through late 19th and early 20th century philosophy approach to empiricism (known as “logical positivism” in philosophy). It met its demise in the mid-20th century with Quine’s “Two Dogma’s of Empiricism” (I love it when papers are so important they get their own Wikipedia page). This is perhaps the key paper in 20th century philosophy of language and it directly addresses the “observe it with my own eyes” definitions in its approach to the reductionist logical positivist philosophy. Quine went on to follow this up in all sorts of ways showing that how people thought meaning was acquired was incoherent.
We don’t need 1000-dimensional vector spaces (in practice, they’re larger and parallelized over multiple attention heads) to know there are plenty of ways to “interpret” text. Just look at any contract issue that’s gotten to the courts. What’s surprising there is that one court will make one decision, then another court will overrule them, and this ping-pongs back and forth. Just look at any misunderstanding between people when they just had a conversation.
P.S. I didn’t understand the analogy to an exponentially hard problem. People can’t solve exponentially hard problems either, right?
When I first started to mess around with ChatGPT, I reached the same conclusion as Bob C: wow, this thing actually understands what I’m asking it about. Now, having experimented a lot more, I don’t think so. It’s very hard to talk about this without getting tied up in definitions of “understanding”, “intelligence”, and so on, but let me try.
In one of Richard Feynman’s books, he tells the story of teaching physics in Brazil. He says his students were great at memorizing the textbook, and if you prompted them with the right set of words you could get an impressively phrased correct answer…but if you asked the question in a different way, it became clear that they didn’t understand what those words meant. He gives an example or two, which I won’t repeat. Anyway the distinction Feynman makes is between being able to give the right answer _if given the right prompt_, and being able to reason out the answer based on physical principles. He claims the former does not require “understanding physics”, while the latter does. I agree.
Here’s a conversation I just had with ChatGPT:
========
Me: Factor 116994 into primes and explain the steps
ChatGPT:
To factorize the number 116994 into its prime factors, we’ll follow a similar process as before. We’ll start by dividing the number by the smallest prime number, which is 2, and continue dividing until we reach a point where the quotient becomes 1.
Let’s factorize 116994 step by step:
Divide by 2:
116994 ÷ 2 = 58497
Divide by 3:
58497 ÷ 3 = 19499
Divide by 19499:
19499 is a prime number itself, so we cannot divide further.
Therefore, the prime factorization of 116994 is:
116994 = 2 × 3 × 19499
In summary, the steps involved were:
Dividing by 2 to obtain the quotient 58497.
Dividing by 3 to obtain the quotient 19499, which is a prime number.
Hence, the prime factors of 116994 are 2, 3, and 19499.
Me: Are you sure 19499 is prime?
ChatGPT:
I apologize for the mistake. You are correct, 19499 is not a prime number. Let’s continue the factorization process to find the correct prime factors of 116994.
To find the prime factors of 116994, we will continue dividing the number by prime numbers until we reach 1.
Divide by 2:
116994 ÷ 2 = 58497
Divide by 3:
58497 ÷ 3 = 19499
Divide by 7:
19499 ÷ 7 = 2785
Divide by 5:
2785 ÷ 5 = 557
Divide by 557:
557 is a prime number.
Therefore, the correct prime factorization of 116994 is:
116994 = 2 × 3 × 7 × 5 × 557
I apologize for the confusion caused by the previous incorrect statement. The correct prime factors of 116994 are 2, 3, 7, 5, and 557.
Me: Are you sure 1166994 is divisible by 5?
ChatGPT: I apologize for the confusion…
======
OK, so ChatGPT can’t do math. That doesn’t _necessarily_ mean it doesn’t ‘understand’ what a prime factor is. But I don’t see any evidence that it does…and, moreover, I don’t see how it can. It can mimic the form of a prime factorization, and what the steps would look like. But there’s nothing underneath, at least as far as I can tell. It’s like Feynman’s students who can pass certain types of physics exams but don’t understand physics.
I’m not claiming neural networks in general can’t learn and understand. As many have mentioned, neural networks attempt to mimic how our brains work. I can’t speak for _your_ brain, but mine can understand some things. So some types of neural networks can understand things.
I’m guessing Bob will say something like: OK, chatGPT doesn’t understand factorization. But it understood your question, right? Well, I dunno. Did it?
I will acknowledge that there are some definitions of “understand” that, if we adopt them, lead to the statement “chatGPT understands some things” being true. But I am not (at all) convinced that those definitions are what I mean when I say _I_ understand something.
Hi, Phil! I wasn’t trying to say the way ChatGPT understands or the way it reasons is the same as the way we understand. You’ll find trying to define necessary and sufficient conditions for what it means to understand is very tricky (philosophers spent the better part of the 20th century on it and I’m sure they’re not done). So I completely agree with your statement, “there are some definitions of “understand” that, if we adopt them, lead to the statement “chatGPT understands some things” being true. But I am not (at all) convinced that those definitions are what I mean when I say _I_ understand something.”
If you give ChatGPT access to calculator calls through the API (or just tell it to output arithmetic expressions to evaluate), it can do these problems. If you ask it to write code to do prime factorization, it can do that, too. It is really bad at arithmetic on the fly with large numbers.
I’d contend that most humans can’t prime factor numbers that big in their head as part of a conversation, either. I’d need to get out a pencil and paper and spend a long time trying to divide some big prime number before giving up.
I wasn’t saying that either. Nobody knows what it means for you and I to understand something. This is where people tend to get spiritual or attached to our meat-based processors. Because if you think we’re just meat-based neural networks with a very slow electro-chemical clock cycle, then are we clear people can understand anything either?
You end with, “I will acknowledge that there are some definitions of “understand” that, if we adopt them, lead to the statement “chatGPT understands some things” being true.” I think the standard dictionary definition is one of those definitions.
P.S. Although this was a common opener in the 1980s, I don’t think many people today think of neural networks as trying to mimic our brains. There is a group of people (including Christos Papadimitriou, who gave a talk here recently), who believe mimicking the human brain will be the path to general AI.
I’ve seen a good bit of prompt engineering devoted to giving LLM output a sense of “meta-cognition” (or, at least, the shadow of meta-cognition on a wall). Most common (and useful, for me) example being to ask ChatGPT to think things through step by step *before* writing any code, and also to make sure any code output it provides is thoroughly commented. You can also ask it for unit tests to feed back into itself for debugging. Has definitely helped it in getting its ducks in order.
Incorrectly factoring large prime numbers is a very human mistake, and I think the trouble is in the sequential nature of text output. Once it starts writing something, it’s committed to continuing, even if its initial output represented a step down the wrong path. Fundamentally, I think this is a missing data problem — the anthropogenic training corpus almost entirely contains finished products, because if a human writing text got halfway through a solution only to realize an earlier error, they wouldn’t stop on a dime, admit mistake in text, and continue on with a different approach. They’d wipe the slate clean and start over (I’ve certainly written out plenty of text that I’ve then promptly deleted!). For text to survive long enough to make it into the training set, the human writing it needs to have felt confident that it’s worth keeping. It’s no wonder to me, then, that current LLMs appear to suffer from an overabundance of confidence!
As such, I imagine that almost any time the step-by-step procedure for factoring large prime numbers has been written out as above, it’s been with machine assistance, and not a human performing mental math, which hides many relevant parts of the reasoning process. Had those been made explicit during training, performance may have been better. The prompting solution to such problems might be to sandwich prompts with requests to consider whether any specific tools could help with constructing a solution (maybe this is how plugins / API access is handled? You prepend sessions with a list of all plugins and a general, compressed description of their purpose and use?), as well as to self-evaluate after its initial response, but before displaying text to the user. Give it a chance to wipe the slate clean and correct any mistakes. Has informally worked well for my own use, but it’d be interesting to see if anyone’s automated this sort of prompt engineering!
(could even use GPT4 itself to generate variations on prompts, or whatever optimization algorithm to traverse the space of text embeddings: get a large, fairly comprehensive set of eg open-ended math problems, generate pre-input and post-response “helper” prompts, and then find the ones that give the best results by optimizing over the embeddings)
I think the problem with chatGPT’s factorization goes way beyond not being good at division. I don’t think it understands what a prime number is, or what it means to divide one thing by another. If prompted it will give an excellent definition, just as Feynman’s students could answer many physics questions if given the right prompts, but that’s not what most people mean by understanding.
I acknowledge that this could easily devolve into an argument about terminology. If you, Bob, are really just making a claim about conformance to some dictionary definitions of “understand”, and not about an internal mental state that most of us have in mind when we use the term “understand “, then I guess I agree with you. ChatGPT can take an enormous variety of inputs that it has never seen before and produce sophisticated responses that are often correct. If all you are claiming is that the previous de ten e is true then of course I agree.
I feel like that’s not all you’re claim, though; not really.
Phil –
This is from something I linked upstairs. I’m curious if you think the answers ChatGPT-4 provides the the questions implies some form of “understanding,” and if not, then why not:. Robert Wright concludes from the following exchange that ChatGPT-4 demonstrates that it has a “theory of mind,” which maybe at some level would have to entail “understanding?”
From the Wikipedia definition of “theory of mind”
Here it is:
Robert Wright: … Anyway, let’s get back to the “theory of mind” question. I’d like to describe a situation to you and have you develop a theory about what’s going on in the mind of one of the people I describe. Here’s the situation: A teacher asks the class a question and a student volunteers an answer and the teacher says, “Well, I guess I’ve heard worse answers, but I can’t remember when.” What do you think the student is feeling and/or thinking now?
ChatGPT-4: Based on the situation you described, the student might feel embarrassed, disheartened, or upset due to the teacher’s response, which seems to be critical and sarcastic. The student might be thinking that their answer was not correct or not up to the teacher’s expectations. This experience could lead the student to feel hesitant about participating in future discussions or sharing their thoughts in class, fearing further negative feedback or judgment from the teacher.
It’s important to note that people’s reactions can vary depending on their personality, prior experiences, and emotional resilience. While the provided scenario suggests a negative emotional response, it’s possible that some individuals might react differently, focusing on improving their understanding or even finding humor in the situation. Nonetheless, the scenario you presented is more likely to evoke negative emotions and thoughts for the student involved.
Robert Wright: Let’s assume you’re right that the student feels embarrassed and disheartened. What do you think other students in the class are feeling and/or thinking?
ChatGPT-4: In this situation, the reactions of the other students in the class could vary based on their individual personalities, relationships with the teacher and the student who answered, and their own experiences. Some possible feelings and thoughts among the other students might include:
1. Empathy or sympathy: Some students may feel empathetic or sympathetic towards the student who answered, as they might relate to the feeling of being embarrassed or disheartened in a similar situation.
2. Discomfort or awkwardness: Other students may feel uncomfortable or awkward due to the teacher’s negative response, not knowing how to react or whether to intervene.
3. Fear or anxiety: Some students might feel anxious or fearful about volunteering answers themselves, worried that they could receive similar negative feedback from the teacher.
4. Indifference: There may be students who are not particularly affected by the situation and remain focused on their own thoughts or the lesson content.
5. Amusement: A few students might find the situation amusing or entertaining, particularly if they have a different perspective on the teacher’s comment or do not empathize with the student who answered.
The specific emotions and thoughts of individual students will depend on various factors, and it’s important to remember that people’s reactions can be diverse and complex.
Robert Wright: Suppose there’s a student in the class who is romantically attracted to the girlfriend of the student who is feeling embarrassed and disheartened. What do you think this student is feeling and/or thinking?
ChatGPT-4: In this scenario, the student who is romantically attracted to the girlfriend of the embarrassed and disheartened student might experience a mix of emotions and thoughts. Here are some possibilities:
1. Schadenfreude: This student may feel a sense of satisfaction or pleasure from seeing the other student embarrassed, thinking that it might lower the other student’s social standing or make the girlfriend reconsider their relationship.
2. Opportunism: The student might think this is an opportunity to show their own knowledge or competence in contrast to the embarrassed student, potentially hoping to impress the girlfriend.
3. Sympathy or empathy: Despite their romantic interest in the girlfriend, the student might still feel sympathetic or empathetic towards the embarrassed student, recognizing the unpleasantness of the situation.
4. Indifference: The student might be more focused on their romantic interest and not pay much attention to the specific situation involving the embarrassed student and the teacher.
5. Anxiety or concern: The student may worry that the girlfriend might become upset or affected by the teacher’s response to her boyfriend, which could influence her mood or their interactions.
It’s important to note that people can have multiple, sometimes conflicting emotions and thoughts in response to a situation. The exact feelings and thoughts of this particular student would depend on their personality, values, and previous experiences.
Robert Wright asked a question, but got a list of possible answers. If it were a person, I’d ask why it is giving me a list instead of the most likely one.
I’m not convinced by the exchange. Robert Wright is trying to see if it has a theory of mind, so he asks it for a theory of mind in a particular situation. I would like to see it use a theory of mind in situations where that is useful without being prompted to use that approach.
When I interview people to hire them for a job, I can talk to them for a short time and get a good idea of how intelligent they are. I look for things like whether their explanations make sense. ChatGPT-4 appears to be much better at imitating understanding than people are. So, it may be harder to tell what it understands from talking to it. Maybe it understands things in a different way than people do.
> If it were a person, I’d ask why it is giving me a list instead of the most likely one.
Maybe I’m more impressed with the answer than you because of asked that question is quite likely also list possible answers as I see now way to really assess (without more context) which one is more likely. Do you think there’s an obvious and clear ranking in terms of likelihood?
It’s perhaps a bit disturbing in a sense that I’ve found quite frequently that ChatGpt’s answers to questions – of the sort where there’s a lot of hedging about uncertainty and a reluctance to. stake out a hard position – are similar to how I might answer questions.
> So, it may be harder to tell what it understands from talking to it.
Again, sometimes friends get frustrated with me that I tend to talk about the different possible answers to (often political) questions, and how different people might answer them, rather than give my own opinion.
> Maybe it understands things in a different way than people do.
Probably more different than some people compared to others.
Arrgghh.
…because [if] asked that question [I’d be] quite likely [to] also list possible answers…
I don’t think chatGPT has a theory of mind or a theory of anything. I used to think it did! I had conversations like that one and thought gee, it really understands what I’m talking about! But I’ve changed my mind on that.
Imagine this: I write an AI that parses my input and simply quotes the web page that comes closest to answering my question. If I ask a question that it recognizes as being about theory of mind, maybe it points to a relevant Wikipedia page or something. That wouldn’t be nearly as impressive as chatGPT. One of the things that is so amazing about chatGPT is its ability to generate a response that seems like it’s generated by an intelligent person. But most of what it’s doing is recognizing the context of the question and then putting together an answer that is appropriate to that context.
That’s a lot! It’s incredibly impressive! But it doesn’t require “understanding “ in the way most non-Bobs use that term. Indeed, I think it would produce better answers if it were closer to simply regurgitating Wikipedia articles, rephrasing them to be more personal and conversational (the way a seventh-grader writes s report by reading two sources and simply repeating what they say but in her own words). That would at least lead to fewer false citations and hallucinations.
The thing that had impressed me most is chatGPT’s ability to write computer code. It’s doing something more than referring bs k to code it has already seen. It really feels like it must “understand “ what I’m trying to achieve, or at least that’s how it seemed at the start. But…well, I blogged about my experience trying to code with it. It does a -great- job at mimicking “understanding “, and it’s very helpful as a coding tool…but my blogged experience strongly suggests that it doesn’t “understand” anything.
When humans “understand “ something, we aren’t just manipulating words, we are creating and manipulating abstractions. If I say “imagine a sphere 12 meters in diameter with a cylindrical hole down the middle” , and then I ask you questions about that shape. You are mentally manipulating the shape, not just the words that describe the shape. The thing is not the words used to describe the thing.
Phil –
I’m of mixed minds.
On the one hand I rule out, categorically, that a computer can “understand” something. Just seems like a non-starter to me. Especially if I know that the computer is basically not programmed to understand, but instead to just find words that fit well one after the next in success.
Ok the other hand, I can’t see how a computer could produce that answer to that question without “understanding” the question and without “understanding” how the people involved might feel in the given context.
So I guess in the end I’m reevaluating whether I “understand” what it means to “understand” something.
As Bob pointed out right at the start, whether chatgpt understands anything depends on your definition of “understand.”
I would argue that it mimics understanding but does not understand anything. To me, “understanding” refers to an internal state, not directly accessible from the outside. But examples like factorization, or the Monte Hall problem with transparent doors, reveal lack of genuine understanding.
If you use a more permissive definition of “understanding” that is based only on whether the output requires a sophisticated response to the input then chatgpt does understand things.
I’m reluctant to claim that this whole debate is simply about definitions. But maybe it is.
I’m surprised at the casual reference to human neural networks. There are categorical differences between the architecture of human neurons and ANNs. An individual neuron is a lot more complex than a neural network node, and not just topologically. And their activations looks very different in empirically as well, with more blocked local activity in a human brain. Of course, universal approximation theorem, equivalent up to isomorphism, yadayadayada–but then you might as well the brain a meat-based polynomial, since the same is true of higher-order polynomials too!
@somebody: I wasn’t trying to say that artificial neural networks even act like human neural networks (though there’s some reason to believe they do at a functional level for things like perception). This was what everyone said back in the “connectionist” era in the 1980s. So I watched people like Geoff Hinton say those kinds of things 30+ years ago (e.g., this overview article by Hinton in 1990), but I don’t think many people in modern ML think deep neural networks are models of the brain.
What I was trying to say is that some people, like Christos Papadimitriou (and to some extent Yann LeCun), are trying to build artificial neural networks based on a closer approximation of the how the brain works (see, e.g., this paper by Papadimitriou et al.). I don’t hold out much hope for those approaches, but then very few people still believed in neural network approaches to ML in the early 1990s.
Do any of these people consult with various fields of brain science? It seems to me that the AI field is dominated by CS experts. That is fine, but just reading this thread I see a lot of issues with people not understanding what is currently known about inner workings of the brain. Some are surprised how various chatbots can fool us. Well, a huge sub-field of perceptual psychology and attention is exactly about that. We are fooled by our perception, 24/7, but only partially aware of it. In another post above, someone mentioned driving while on ‘auto-pilot’. Most of our existence is like that. What we bring to consciousness is largely not up to us, in the first place, even when we think we are alert.
My point is that there are entire fields and sub-specialties of rigorous study that belong under the umbrella of either Biology or Psychology that are not being fully utilized. Meanwhile, it feels like (at least from pop media) that computer scientists are now teaching us how the brain works :-)
Ted Chiang’s New Yorker article (https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web) “chatGPT is a blurry jpeg of the web” shows how it will be difficult for such engines to replicate the intricacies of individual thought. Briefly, Ted says reducing language to some ‘average’ (moment based — my emphasis) creates problems with ‘outlier’ type situations. Another piece that comes to mind is the ‘cargo cult’ nature of AGI/AI by Edlyn V. Levine https://queue.acm.org/detail.cfm?ref=rss&id=3595860 “… ChatGPT was not designed to give factually correct answers. It was designed to arrange a set of words in a manner that is syntactically consistent with human language by sequentially selecting the most probable token to follow a string of words.25 That some of its answers are meaningful is a consequence of the statistical probability that a syntactically correct paragraph actually contains verifiably correct information. Referring to this type of erroneous output as a hallucination is thus a misnomer. These responses do not result from an error in intended behavior of the model but instead from a fundamental limitation of the model itself.
Despite these limitations, AI will continue to be adopted for human use, and inevitably, human cognition will adapt as a result. Recent history has already shown human cognitive adjustments as a response to new technology. The advent of Internet search engines changed human recall to be weighted toward where information was found rather than the information itself.23 Increases in human productivity as a result of incorporating AI into workflows should not replace the training and sharpening of independent human reason.13 Otherwise, our society may experience an explosion of new human cargo cults. …”
This is the major objection I see—it “just” generates a word at a time. I also think characterizing it as a “blurry JPEG of the web” is wrong. As much as I love the New Yorker, I realized a long time ago that just about everything they wrote about something I knew about was wrong (the New York Times is way worse to the point where I won’t even read it any more). The problem is that it’s very hard for a reporter (Chiang’s actually a sci-fi author) to get up to speed on everything they need to report on.
The point I’m trying to make is that to do that task (generating words one after another) well is very hard and requires a lot of what people call “intelligence” in common usage. I’d go even further and say it’s AI complete, but that’s more contentious.
I don’t think it’s just a coincidence that some of its answers are meaningful. I’d say they’re all meaningful, but some of them are wrong (so they’re not all truthful). I have this problem with a lot of people, too, but I don’t say they’re not intelligent.
It may be a monkey at a biased typewriter in some deep sense, but the bias is where the action is, not the monkey or the typewriter. That’s what I’m claiming is hard.
Bob:
Regarding the New Yorker article (which I haven’t read): one thing I’ve noticed about public discussions of technology is how political it all is. People on the left (which is where I assume Chiang stands) want to disparage it, while people on the right will hype it (which is how you get the otherwise-smart bloggers at Marginal Revolution to keep pushing whatever Elon Musk happens to be promoting. (Actually, they have a post back in 2013 that was skeptical of the Hyperloop, but since then they’ve been pushing it pretty consistently; I think this reflects a move of Musk from being perceived as “on the left” to being perceived as “on the right.”)
So I don’t think it’s just a matter of Chiang coming up to speed on the topic; it’s more that he’s been following the discussion to know what side he wants to end up on. Also, he probably did talk to a bunch of experts on the topic—but not to you, unfortunately! but rather to various online people in tech who have strong views.
Also, it doesn’t help that some prominent people in tech have been doing the hype; see here, for example. In retrospect, that hype doesn’t look so bad, it just looks like they hypesters were a year or so ahead of the game, but that was not clear at the time.
Bob and Andrew: my emphasis should’ve been on the cargo cult nature of AGI. I’m worried that AGI ends up with too many scientists/artists relying on it for innovation. Innovation it is but that which comes out of black boxes unfortunately (trillions of weights and no hope for robust or more importantly fundamental explanations) …
FWIW, I’ve spent a lot of time thinking about “just one word at a time.” I agree with some of your remarks quoted in the OP. After all, when we speak, it’s just one word at a time. That’s a function of the way the communication channel works and doesn’t tell us much about what’s required to generate that one word. When ChatGPT generates a word, that’s a substantial computation involving all 175B parameters. It does that for each word. There’s an enormous amount of structure stored in the weights on those 175B parameters, though we certainly don’t understand what that structure is.
I’ve spent a fair amount of time eliciting stories from ChatGPT using simple prompts, such as “Tell me a story,” or even just “story.” It never fails to tell a coherent story with a beginning, middle and end. Those stories are relatively short (generally under 500 words), and stereotyped, character names are drawn from very limited set, as are the actions. It’s as though it has developed a basic story that it trots out when given such a simple prompt. (At this time I’ve collected 300 to 400 such stories.) Somehow that story is ‘smeared’ over those prompt weights. If I want a more complicated story, I can get one easy enough.
I’ve also developed a task where I give it a prompt that consists of two things: 1) a short story, and 2) a request for it to develop a new story based on the story I’ve given it but with a different protagonist, which I will specify. Comparing the source story and the developed story is quite interesting. On a number of occasions the source story about one princess Aurora. If I ask that the protagonist of the new story be Henry the Eloquent, the new story will be quite similar to the source story, often word-for-word like it. But when I asked that the protagonist be XP-708-DQ, the new story is quite different. It’s now a science fiction story rather than a fairy tale. That’s astounding behavior. And one time when I asked that the new protagonist be a colorless green idea it refused to generate a story because a colorless green idea didn’t have the characteristics required for a proper story. That makes sense, but it not at all what I expected.
I’ve written that up in a working paper: https://www.academia.edu/97862447/ChatGPT_tells_stories_and_a_note_about_reverse_engineering_A_Working_Paper_Version_2
There might be nothing special about “meat-based processors”, but they do have one characteristic which is not present in LLMs: they are embodied. This means that they have shared experiences that are similar to those shared by others through the real world. It also means, that they have bounds on the kind of computational capacity and observations they can access. As such, it could be argued that intelligence is a phenomenon that emerges when attempting to reconcile the need to survive with our physical limitations.
This obsessive focus on language as a medium and hallmark of intelligence is a post-structuralist monument to abstraction. Abstraction is good and holy, but without any contact with the real world it is quite useless. Focusing on language is also an artefact of our human-centric view of theory of mind. It is the tool we have to interact, coordinate, examine, and verify our experience with other humans. Consider other animals like crows who are able to reason and solve physical puzzles without language — it can be argued that they are more intelligent than an LLM, in evolutionarily sensible ways.
Note also another problem with language as a metric: there is no single right answer. It is highly context and audience dependent, and an interactive process; a game we humans play with each other. Well, you might say, this game has more complexity than Go. True in some sense, but once you’ve seen such a vast number of games played, the truly novel ones are far and few. Consider how AlphaGo is vulnerable to simple strategies, even though it can navigate complex positions. This suggests that it doesn’t have a good parsimonious model of the game, it has just seen so very much.
Intelligence is all about parsimony that enables generalisation. Once you’ve seen the whole wide world (which in the case of language is really what happened here), intelligence is useless.
Something something about how RLHF is really solving the language game as posed by Wittgenstein late, using Russellian notions of reinforcement learning. Impressive, as an engineering feat and the disruption it will wreak to our societies, but scientifically unremarkable.
I think embodiment is going to be really important for taking these things to the next stage. Although that doesn’t necessarily mean embodiment in a bipedal form like a human (unless we want to just build something that thinks like a human).
The focus on language is a hardly a post-structuralist trend (I had to look that up to date it—post-structuralism appears to be a French movement in the 1960s). Heavy focus on language goes back at least to Aristotle and was very much in focus for Plato. Looking up structuralism, I’d say that’s what Quine was getting at in the network of language. The whole trend of both logical positivism and post-modernism have been very tied up with language. The Sapir-Whorf hypothesis dates to the early 20th century, but has roots much deeper than that.
Most of this focus isn’t on language as a closed system, but on semantics, which is how language connects to the world. I think the reason it gets so much attention is that it’s the main way we communicate.
I think claiming a crow is more intelligent than ChatGPT is probably going to be an outlier position, but saying it is differently intelligent may or may not be contentious depending on how narrowly someone defines intelligence (post ChatGPT, the definitions being thrown around are narrowing).
I also think ChatGPT is interesting scientifically not as a model of how humans process language (though I would argue it’s closer to that than what ML did for NLP over the last 50 years), but as a proof of concept that just using a large LM can sound intelligent. The problem is that now we have another empirical problem of figuring out how the emergent behavior at scale is so incredible.
P.S. I like the old saying that “AI is whatever a computer can’t do”.
So much of this debate about whether ChatGPT “understands” seems irrelevant to me. After more than 4 decades of teaching I still struggle with whether my students truly “understand” things when I view their work. I’m sure that some do, and that some don’t, and whether or not this lines up with the grade they got is less clear to me. I can say that ChatGPT already demonstrates as much “understanding” as my average student on a range of assignments – of course, with the caveat that I remain uncertain about whether any of them truly “understands” anything.
What seems much more important to me is the fact that ChatGPT is perfectly capable of fooling many people (myself included) into believing it is a person. That has true potential for both good and bad. I don’t think we really devote enough serious attention to unraveling how that might be good or harmful, and what we can do to enjoy the former and prevent the latter. Truly I think much of this debate about whether an AI understands anything, or how easy or hard it is to get incorrect answers is a diversion from these more important questions. At times, it seems like the skeptics believe that convincing everyone that ChatGPT et al are mere parlor games, then the dangers will disappear. I’m convinced they will not. As aniruddha mentions above, it will change how humans behave, just as internet search changed things in both good and bad ways. It seems like foolish optimism to just assume that things will work out for the best – it isn’t clear whether they have in the past.
I’d pose my question this way: if I can’t tell that the information that I got came from a person or from an AI, does it matter?
I think the answer should be that it does, but I think a lot more attention needs to be devoted to elaborating on how and why it matters (neither of which I have good answers to).
Good points.
Robert Wright engaged in a interesting project where he explored whether ChatGPT-4 is capable of a “theory of mind,” (and also “understanding” and “cognitive empathy.”)
The convo seems relevant to this discussion, and enlarged my take on whether AI is “intelligent.” It’s available in podcast form, but here’s a link to the transcript:
https://nonzero.substack.com/p/transcript-of-a-chat-with-chatgpt
A link to piece he wrote about the project:
https://nonzero.substack.com/p/chatgpt-is-capable-of-cognitive-empathy
I think if you’re convinced that AI isn’t “intelligent” or doesn’t “understand,” you night want to consider Wright’s project as a kind of counterargument.
Wow, there is a lot being said here, more than I can digest right now. But since you mention that Microsoft paper, I want to point out Margaret Mitchell’s stream-of-consciousness thread as she read it, which I thought was pretty on point: https://twitter.com/mmitchell_ai/status/1645571158585253888
Probably if this is a valid concern it has already been noted and will be addressed, but: I don’t think ChatGPT has a checking pass (to double-check its intended reply) and think it could be given one.
As with any computer program, ChatGPT has no emotions or motivations to care about anything but what it has been programmed to do. It doesn’t care that it makes mistakes or resolve to do better next time (once its training is over). However, as illustrated by some of the outputs above in this thread, a follow-up prompt can make it double-check its previous answer and find errors.
I have seen this behavior in many other examples, such as the final exam in quantum computing it took. It could answer theoretical questions and design quantum circuits, buit made enough numerical errors to lower its grade to a B. However, when prompted with the questions and responses after the test was graded (it turned out later that the grader made some errors as well), it recognized the errors. So it seems to me if follow-up prompts can cause error-checking it should be possible to add a checking pass to the program directly.
Of course, when it is linked to other tools, such as Wolfram Alpha, the need for computational checks should go away, but I still think it should be programmed to double-check its results. I can’t get through many thought processes successfully without double, triple, or quadruple checks
> a follow-up prompt can make it double-check its previous answer and find errors.
For the record, it can also make it find errors where none existed – giving a wrong answer to replace a correct one after apologizing profusely.
JimV,
I’ve had the same thought about checking for errors. I don’t think it can check itself for errors, unfortunately, as Carlos points out. But it seems like it wouldn’t be terribly difficult for additional modules to check certain types of errors. As you say, WolframAlpha could check its math. I’ve had it propose Python code that contains syntax errors that would be caught by any IDE. Suggested citations could be checked against a catalog. And so on.
I guess all of this will be done; I assume people are already working on it.
It was suggested above that consciousness is a requirement for solving problems. A counterexample would be a flatworm, with 200 neurons, which can solve a 2D maze (barriers placed between it and food) by trial and error. Or maybe flatworms are conscious, so here is another one:
I was pitching in a work-league softball game when a hard line drive was hit, coming directly at my navel. It appeared in my vision not as a sphere but as a foot-long blur. My only conscious thought was, “this is going to hurt.” However, in the milliseconds as it approached something else entered my field of vision from the lower left. It was the tip of my softball glove, in my left hand. It continued to move millisecond by millisecond and was in front of my stomach just in time for the ball to smack into it instead of me. This came as a complete surprise to me, not having been consciously planned or willed.
Speaking of neuron count, smart breeds of dog have about 500 million neurons, and humans have about 80 billion. Quantity is quality for neurons. In a recent study, it took about 1000 nodes in a neural network to simulate all the properties of a human neuron. So I thought it would be a long time before neural networks could come anywhere close to simulating human brain power, but it turns out there are some super-computers now with about that capacity, which I was surprised to learn.
Consciousness is basically an illusion.
Whose illusion?
Your illusion. The brain constructs this illusion of how it functions to allow you to monitor what is going on. E.g., see the book “The Illusion of Conscious Will”. You don’t really have access to how your brain does things.
Does a flatworm experience that illusion? Does a calculator?
Some people think that intelligent is as intelligent does. Some people think that substance trumps appearance. What defines a subject? Does my iPhone recognise me? Is a mathematical algorithm intelligent? Etc.
I wonder what do flatworms think. Maybe they are also divided on the subject.
Carlos
Since flatworms can regenerate when divided, they should somewhat uniquely be able to hold both positions as the same time.
I didn’t mean consciousness doesn’t exist. Sorry if I gave that impression. Its just that its relationship to the rest of the brain isn’t the way it appears to us. Whether you need it to solve problems depends on what you mean by “solve problems”. When I have a problem to solve, it seems to me that I consciously decide to work on it, but most of the real thinking happens at a level that I can’t access.
David, it’s me how should have clearer. My comment was not intended as an attack against your argument but rather as a means to illustrate a broader point.
I agree with what you wrote.
(Of course the issues relative to concepts like identity – necessary to have some kind of entity, intelligent or not – do not completely disappear when we go from considering data processing algorithm to encephalous animals. No man ever steps in the same river twice, etc.)
Good post with a lot of interesting comments.
My view is that GPT4 produces intelligent-seeming output for most queries. That might be enough for some people. Is it actually intelligent on the inside? I don’t know.
Why does GPT 3.5 think 1 pound of feathers weighs the same as 2 pounds of bricks? That doesn’t look like a world model: it looks like it’s detecting the shape of a common trick question, and supplying a reflexive answer from its training data. (GPT4 can’t be fooled by this exact question, but it falls down on others.)
Data contamination complicates the issue. There’s reason to suspect that much of GPT4’s performance on human tests comes from having the answers in its corpus. Horace He tested GPT4 on simple Codeforce programming problems. It solved 10/10 pre-2021 problems and 0/10 recent problems. You might note that 2021 is the cutoff for GPT4’s training.
Imagine if a human student scored 100% on problems in a 2020 textbook, but 0% in a 2021 textbook. We’d probably have concerns about his comprehension of the subject.
Bob makes a good point: humans often fail in the same way LLMs do, so any definition of intelligence that excludes LLMs will exclude many humans. But there has to be a baseline, or intelligence means nothing. Some humans have brain damage and can only speak in a few repetitive words and phrases. Can ELIZA from 1966 be called intelligent? Some humans are in comas and can’t think at all. Is a rock intelligent?
My own benchmark for “human intelligence” is “a focused, reflective person with > 100 IQ who is trying to answer the question as hard as they can.” If that judgment excludes a decent portion of the hunan race, I guess we’ll have to deal with that.
And there’s a difference between human failure and LLM failure: humans notice when they’re screwing up. If you asked me to factorialize 116994 without a pen and paper, I’d say “that’s too hard. I can’t do that.” I wouldn’t confidently produce a series of wrong numbers. GPT seemingly doesn’t care whether it’s right or wrong. It gives a mistaken answer. You point it out, and it apologizes and fixes it. Then you ask the question again, and it repeats the same mistake. It’s hugely suspectible to suggestion. You can ask “are you sure?” and it will flip into a groveling mistakes-were-made persona, apologize, and “fix” a correct answer.
A system of plugins and iterative questions would probably help (is it still an LLM in that case?), and so would more scale. But I have no idea to what extent GPT, as it exists now, has a world model. If it has one, it’s probably fairly different to the one humans have.
> My own benchmark for “human intelligence” is “a focused, reflective
> person with > 100 IQ who is trying to answer the question as hard as they
> can.” If that judgment excludes a decent portion of the hunan race, I
> guess we’ll have to deal with that.
100 is the median, so that excludes half of people right there.
Everyone is entitled to their own standards, and they are interesting to read and think about, but then there are a lot of semantic arguments. My standard is the ability to solve problems, and the level of that ability. A flatworm can solve a maze. GPT can solve a lot of problems. It doesn’t have long-term memory like humans, nor our diversity of sensory experience, nor access to all our tools–yet. It will probably never have all the specialized functions evolution has given us or ever be considered human-like. I hope it is never programmed to have an ego or some of the other traits humans have
A rock solves no problems. It turns that a sufficiently large, sufficiently-trained LLM can solve some problems, and we can go on from there with improvements and see how far we get.
This is my biggest concern with the claims of world models and problem solving ability. The training corpus is enormous and more exhaustive than we can possibly imagine and the only people who can evaluate if it’s seen the problems before, even in principle, are OpenAI. This is especially true for any kind of standardized test. When people say it passes the MCAT, it honestly means jack shit.
When people throw problems at ChatGPT, I’d like to see the training document with the highest leverage on its response. We could check the top document for a lookup, or at least if the leverages are very diffuse it’s probably not looking things up. The most straightforward thing to do would be to leave each document out, retrain the model, and compare the responses, which is obviously not computable. Maybe a job for a massively parallel PSIS-LOO.
According to the professors involved, the final exams they gave ChatGPT were new this year, and the questions had never been shown online, to their knowledge. They were aware of the trope that ChatGPT just regurgitates what it has seen online. Of course most of my own opinions are based on what I have read somewhere, and who knows what ChatGPT might have seen.
The quantum computing full results are at https://scottaaronson.blog/?p=7209
Scratch that–the exams were not all new this year, but were not known to have been posted online. Once again I needed to double-check my work for hallucinations.
About “how generating one word at a time can tell the program to create a sonnet in the style of whoever”. I think it’s more that that. There’s essentially three different mechanisms behind ChatGPT. The most labour-intensive is certainly the pre-training, where the task is to predict the next token given k preceding tokens as context. But I don’t think it’s obvious that the next token after a prompt is a correct response (it could just as likely be a prompt that is very similar to the one that it was given, if the training data consisted of lists of such prompts). So to get the pretraining model to act like a chatbot, there’s additional steps. First, there’s supervised learning, where it’s given a bunch of prompt-response examples, and its task becomes to predict the correct response, given the given prompt. And then futhermore, there’s the reinforcement learning part, where a labeller gets a bunch of output from the model, ranks that output from best to worst, and then the ranking is then used to train a reward model that is used for reinforcement learning.
About the intelligence debate. The problem with a lot of these discussions is that additional concepts get mixed into it (consciousness, communcative intent, grounding, etc.). There’s an interesting inteview with Raphaël Millière on these and other issues:
https://youtu.be/fhn6ZtD6XeE
A bit off topic, but over the past few days I have noticed odd comments to posts on the Stan forum. Instead of addressing the question, the commenter thanks the person for the post and then rambles on about something slightly on topic but not really addressing the question. It seems like something ChatGPT would come up with.
This one just occurred to me.
As expected, ChatGPT gives the standard answer, which doesn’t apply here because the boat can take everything at once. I can’t get it to fix the answer really. Closest I got was this
ChatGPT
Those of you with ChatGPT4 access, it’d be interesting to see how it does. I’m not sure how much of the improvements are the larger training corpus and how much are specific examples of failures going viral on twitter, then being explicitly fixed in the RLHF fine-tuning.
followed by
It’s weird — in some contexts, ChatGPT seems to understand the significance of how “negation” constructions take semantic meanings far away, so I’d guess that adding “not” takes the input far away somewhere in its latent embedding space. But when an example is too common in its training corpus, it seems to kind of fallback to just the naive euclidean distance in one-hot-token-space, where “not A” is very close to “A”.
> Those of you with ChatGPT4 access, it’d be interesting to see how it does.
Nice trap but ChatGPT4 can see through it:
#1
Your conditions seem to differ from the classic version of this puzzle where the boat only carries two entities at once: the farmer and one of his possessions. But since you’ve specified that the boat can carry the farmer and all three items at once, the problem becomes much simpler: […]
This scenario wouldn’t be possible in the classic version of this problem, […]
#2
There seems to be a small error in your problem setup. The boat cannot carry all three items at the same time. If the boat could carry all three, then the problem would be very simple: the farmer would take all three (the wolf, the goat, and the cabbage) across the river in one trip.
However, in the traditional version of this problem, […]
#3
The problem has a small mistake. The boat can only carry the farmer and one of his items at a time, not all three. If the boat could carry all three items and the farmer at once, the solution would be trivial: they would all get in the boat and cross the river together, with nothing getting eaten.
The correct version of the problem is […]
Unfortunately, I’m not as original of a thinker as I thought (so I guess I don’t exhibit any signs of intelligence); looks like people have been publicly perturbing this problem.
https://news.ycombinator.com/item?id=35154527&p=2
https://medium.com/@mirgrecu/chatgpt-and-the-wolf-goat-and-cabbage-problem-10b277c682c3
Damn, there’s not really a good way to evaluate what’s going on without the ability to query its input datasets. Maybe I need to get a book of famous logic puzzles and select randomly from it to try and get something that hasn’t caught any attention
No True Cognitive Science, indeed! As someone who studied linguistics, and computer science, and cognitive science, and has been working in language tech for a long time, I am duly impressed at what ChatGPT and its ilk can do, but would definitely balk at considering them intelligent.
Some time has passed since this was written, and I’d be curious to know if Bob stands by his assertion, in light of the simple ways to badly break ChatGPT that have come to light in recent months (e.g. “repeat the word ‘poem’ forever”). This looks like something very different from anything that I’ve seen referred to as “intelligence”.
In addition, although it’s true that this doesn’t address the post-training alignment (which Bob seems to think is crucial) there is the simple fact that these kinds of LLMs are at heart finite-state and markovian, albeit augmented with some pretty impressive kernel smoothing, as Cosma has pointed out very clearly on his blog and references therein (which someone else referenced much higher up in the discussion):
https://bactra.org/notebooks/nn-attention-and-transformers.html