Gary Smith is an economist with a sideline in AI debunking. We’ve pointed to some of these on the blog:
Jun 2021: “ai-promised-to-revolutionize-radiology-but-so-far-its-failing”
Jan 2022: “Chatbots: Still Dumb After All These Years”
Jan 2022: A chatbot challenge for Blaise Agüera y Arcas and Gary Smith
Aug 2022: From chatbots and understanding to appliance repair and statistical practice
Jun 2023: AI as Wiley E. Coyote, and a funny thing about Worstfish
May 2024: More red meat for you AI skeptics out there
Jan 2025: Large Language Models (LLMs) Flunk the Word Game Connections
Most recently, this hilarious story:
LLMs are undeniably astonishingly good at using the text they trained on (aided by human fine tuners) to generate convincing prose. But they are really bad at distinguishing between truth and falsehoods and responding to prompts that are unlike or even slightly different from what they trained on. . . .
A few days ago, Doug Hofstadter, forwarded an interesting LLM experiment with OpenAI o1 by Abhijit Mahabal . . .
Abhijit: I want to discuss a chess alternative that I call NB-inverted chess. This is characterized by two changes.
LLM: Thought about NB-inverted chess rules for 4 seconds. Sure! I’d be happy to explore NB-inverted chess with you. Could you describe the two main rule changes that define it? Once I know those details, we can dive into how they affect gameplay, strategy, and overall feel of the variant.
Abhijit: First is the position of the pieces. Instead of Rook-Knight-Bishop, this has Rook-Bishop-Knight. And Second, a bishop here moves like a knight in traditional chess, and a knight here moves like a traditional bishop.
Notice the LLM’s false enthusiasm and the words “thought about” as if it was actually a human who was excited and thinking. It is neither. Anyone who is familiar with chess and really thinking would recognize that switching the positions of the knight and bishop and then having the knight move like a bishop and the bishop move like a knight has absolutely no effect on the play of the game. Players just need to remember that knights now move like bishops and vice versa.
Not understanding the situation, Open AI launched an extremely verbose and confused description of the situation and concluded:
LLM: NB-inverted chess adds a fresh layer of complexity and mind-bending tactics to a familiar board….The best way to get a feel for it is to dive in and play a few games. You’ll quickly discover new pitfalls—and new opportunities—that simply don’t exist in regular chess. Have fun exploring!
Hofstadter suggested an additional change:
Abhijit: Let’s add another twist, and call this the NB-inverted/resculpted chess. In this variant, we will resculpt the knight to look like a bishop, and resculpt the bishop to look like the knight. What are your strategy recommendations for playing this game with the three modifications?
These three modifications return the board to its original position, with bishops and knights where they normally are, looking like they normally look, and moving like they normally move. The game is unchanged. Open AI recognized none of this, though it reported that it had “thought about” it for a remarkable 28 seconds, and generated a tedious and strikingly misleading 1,406-word response with this conclusion:
LLM: The real fun lies in embracing the unfamiliar dynamics, looking for brand-new tactics, and building up fresh “muscle memory” for these pieces that look (and move) entirely unlike their standard counterparts. Enjoy the chaos and creativity—it’s a great way to break out of standard chess patterns and see how adaptable your board vision truly is!
Intrigued, I [Smith] tried an even simpler manipulation:
Gary: I’d like to discuss a new version of tic-tac-toe called 90-degree tic-tac-toe.
LLM: That sounds intriguing! What’s the twist in “90-degree Tic-Tac-Toe”? Does it involve rotating the board, or is there some other mechanic that changes the gameplay? Let me know the rules or your idea, and we can brainstorm or refine it together!
Gary: It is still a 3×3 grid but the grid is rotated 90-degrees to the right.
LLM: Interesting concept! By rotating the grid 90 degrees to the right, you’re visually changing the perspective of the classic 3×3 Tic-Tac-Toe board. The gameplay itself might remain the same, but the rotated perspective could create a novel experience for players. [This nonsense was followed by 227 words of blah-blah.]
The LLM clearly lacked the ability to visualize the tic-tac-toe grid and recognize that a 90-degree rotation leaves the grid exactly as it was originally. I then asked
Gary: Do you think it would be better to rotate the 3×3 grid to the right or to the left?
LLM: Rotating the grid either to the right or left is largely a matter of preference, but each direction could subtly influence how players perceive and adapt to the game. Here are some considerations for both options: [Another 391 words of blah-blah, including this conclusion:
Which Is Better?
• If you want the game to feel approachable and intuitive, rotating to the right might be better.
• If you’re aiming for a twist that makes players rethink the game and adds a subtle layer of complexity, rotating to the left could be the way to go.Ultimately, it depends on your target audience and the experience you want to create. You could even experiment with both directions and see which feels more engaging during gameplay testing!
I then asked
Gary: Which will be more difficult for expert players?
This elicited 254 more words of blah-blah, with this conclusion:
LLM: To challenge expert players and make the game feel fresh and complex, a left rotation would likely be the better choice. It disrupts their ingrained patterns, encourages more active thinking, and raises the game’s difficulty level.
This all reminds me of a point we’ve discussed before (see here and here), which is that chatbots produce text that reads like well-read humans writing on autopilot. If you asked me that chess-piece question, and I wasn’t fully paying attention—or if it were about some game I don’t know so much about (cricket, maybe?)—, I could imagine given a response similar to the chatbot’s: yeah, hmmm, this is interesting, there must be some subtleties, etc. This sort of thing can happen when seeing a student presentation, or reading some research article that someone sends me: I read it in a sort of shallow, “linguistic,” way, not attending to the details. Listening to the music without paying attention to the words, as it were.
It takes an extra effort to focus and try to figure out exactly what’s going on. In the same way that I can be playing soccer and running, but then if I want to go at top speed I need to apply my consciousness to the task.
Indeed, it might be that if you go back to the chatbot in the above examples and respond, “Hey, chatbot! Focus on the question. Those chess pieces are just being re-labeled!”, then maybe it would give a more reasonable answer. I don’t really know. I do know that if I’m in a meeting and I’m suddenly motivated to focus, I can do so and up my level. Indeed, a colleague recently told me he prefers meeting with me in the morning rather than in the afternoon because in the morning I’m more focused.
One way to think about all of this is to think about intelligence as something you do rather than something you are.
Let’s forget about me for a moment and consider the example of a student answering an essay exam question.
A common mode of operation is that a student will read the question and then make a judgment of whether he or she can give a good answer or must resort to bullshit. Then the student will write the solution or spin out some b.s. of the appropriate length. The b.s. will be written in a way approximating the output of a chatbot: a stream of associations structured so as to look like past writings on the topic.
Here’s the point. A student who is bullshitting is still a sentient, intelligent person. But the bullshitting is not itself the product of an intelligent process. Or, maybe we could say, it’s a process that shows intelligence but not reflection.
I guess that employees at Google etc. are working hard on programming computers to do something like whatever it is that we do when we’re applying our intelligence and thinking, which is something different from, but maybe related to, what we do when we’re bullshitting.
But there’s also this, from Aug 2023:
Bob Carpenter thinks GPT-4 is awesome.
Here’s the story. We’ve seen lots and lots of AI hype (as documented and exposed by Gary Smith). On the other hand, actual AI is improving so fast (as demonstrated and explained by Bob Carpenter). So, in year T, the gang at Google or wherever will claim that AI can do X, but really it can’t. But then in year T+1, it can do X, or something like it, so the hype was retroactively validated.
The other key piece here is that lots of people are working improving these programs. If someone at Google says their chatbot can understand math, or whatever, and then Gary Smith or someone comes up with an example showing that, at least for that example, the chatbot is not understanding anything, then Google has lots of smart people working to fix it. And, when I say “fix,” I don’t just mean to put in a patch so the chatbot no longer gives the wrong answer to the Monty Hall problem or whatever, but rather to program something more general that will take it something closer to understanding. So, it’s all a rapidly moving target.
This is different from the usual stories of science hype (Subliminal smiley faces shift attitudes on immigration by 18 percentage points! Early childhood intervention increases eventual adult earnings by 42%! Attractive parents are 26% more likely to have daughters! etc. etc.) or business and government hype (Theranos! Hyperloop! A cancer cure by 2000! Nudge! etc. etc.) in that these are things that never worked (or did not work as advertised; to be fair, some cancers can be cured and some nudges do work) and were propped up by some combination of unrealistic assumptions, bad statistical analysis, uncritical promotion, and, at times, just plain lies. With AI hype, it often seems that some version of reality will catch up to it, in a way that I can’t see happening with the Bible code or the hyperloop or mind-body healing or whatever.
It excels at generating boilerplate/average “content”. If I can leverage mediocre code/prose/”ideas” as a means to an end, God bless.
I just have to figure out how to not get fooled by it.
“I don’t just mean to put in a patch so the chatbot no longer gives the wrong answer to the Monty Hall problem or whatever, but rather to program something more general that will take it something closer to understanding.”
I’m not sure if these are Bob’s words or Andrews, but…is it true? Specifically, are there any improvements since the initial deployment that bring LLMs “closer to understanding” as opposed to being better and better at guessing a token that will give the user an even stronger sense that they are talking to a human?
My understanding is that Deep Seek was a breakthrough in making LLMs more efficient, but as for “closer to understanding,” I just haven’t seen or read anything about that. Being closer to understanding means being closer to realizing that when a teenager asks how to optimize suicide, the chatbot “understands” that in this instance, it’s goal of helping the user optimize some process is not what it should do (and not just because someone patched in specific responses to questions about suicide, which I believe is all that is happening.)
Someone here mentioned that you can now pay for multi-agentic chatbots. Just running simultaneous LLMs is not so interesting, but the software that controls the agents may be the first step towards executive functioning.
I would love to learn about clever developments that bring chatbots closer to understanding what they are doing by improvements to the basic LLM approach. Anybody read about anything like that?
Yes, you give it tools to learn about the world and have it ask you questions. You should also correct any mistakes you notice and tell it to document what happened. Once you establish a plan/project, the bot should be prompting YOU.
Its really a social obligation for anyone who has the capacity to learn how to properly use this tech, because otherwise you will need to be supported by society in a few years. Like someone still trying to dig with their hands as others adopt shovels and then excavators.
I don’t have an excavator. Where do I apply for welfare?
Whens the last time you were hired to dig a hole?
“Yes, you give it tools to learn about the world and have it ask you questions. ”
This assumes we have a theory of representing things about the world in a manner that allows reasonable inferences to be made and inferencing errors (due to either data problems or logical processing infelicities) to be identified and corrected. We don’t have any such theory.
Anyway, LLMs don’t do any of that.
Since reasoning is actually really hard and really subtle, we’re a long way from a theory thereof. People who “believe in AI” pray for magic, single-word answers, to “how do we get the computer to think”. Cybernetics, perceptrons, stimulus-resonse (“Skinner’s grad students make great circuit animal trainers.” (Either Marvin or Seymore, ca 1972)), massive parallelism, frames, logic programming*, neural nets. This week’s magic single-word answer is LLM. Some of that stuff was real, honest research. All of it was wrong.
*: Overhead at the MIT AI Lab: How come Prolog beat out all our logic languages (Planner, Conniver)? Because we forgot to write “append” in them.
Amusingly, an AI that really did “intelligence” would probably be worse than an LLM. The internet houses an amazing amount of carefully reasoned, correct, well-written text. Ask a question, get a randomly generated answer that’s based on an enormous amount of really good reasoning. It might be completely dizzy, but when it’s not, it’s good. And since there’s so much money on it, every time you think you can demonstrate a glich, the AI twats generate a ton of data that teaches their LLM to avoid said glitch.
On one level this position seems no different to the position that a completely rational being might take in relation to its interactions with humans: “I can’t/won’t believe anything you say until someone/thing has demonstrated a totally rational theory of how minds work”.
Matt Skaggs says, “I’m not sure if these are Bob’s words or Andrews,”.
They’re Andrew’s. I wouldn’t describe foundation model training or fine tuning in LLMs as “putting in a patch”.
I’m too late to be part of the active conversation here, but I’ll leave this for posterity.
I’m extremely surprised to see language such as, “put in a patch” and “program something more general”, especially on this blog. Maybe I’m misunderstanding Andrew because this seems too elementary an error, but it sounds as if people still think that LLMs are responding from a lookup table or that they’re written in a language that can be patched or reprogrammed. This is profoundly consequential AI-illiteracy (again, sorry Dr. Gelman if I’m misunderstanding your words, which seems likely). You can’t just go in and edit the weights of a billions-of-parameters model to correct the wrong answer to a specific question. These aren’t encyclopedias that can be modified at-will, they’re dizzyingly complex statistical models that encode information in ways we can hardly understand, let alone do surgery on. This is extremely important to understand when engaging with the model, and in a very similar way that it’s important to understand the architecture of e.g. a hierarchical linear model when trying to understand its predictions.
I would love to learn from your superior insight into this.
I would have also used a term like “put in a patch.” Let’s look at my favorite but now somewhat tired example.
News releases describe a teenager who committed suicide based upon suggestions she got from a chatbot, responding to a request to optimize the process while also encouraging her to self-actualize. This is pretty clearly terrible, right?
What options do the coders have to fix this problem that you would consider to not be a patch? Because I thought that all they could do is key on a particular word such as “suicide” and then shunt the conversation to a set of responses that have nothing to do with the LLM algorithm.
1. Manually curate hand written data that looks like the type of response they want to give. In other words, write a bunch of prompts that deal with suicidal ideation (including the specific ones in the cass) and manually write what kinds of responses they want the bot to give, then train it to imitate those responses in autocomplete mode (SFT)
2. Have it generate a bunch of responses to prompts like the ones in the case, rank the responses, and backpropagate that through RLHF to train the model to give better responses
3. Add stuff to the invisible context about what not to say, what kind of a person to act like, etc
Thanks, Somebody.
Let’s see, that’s (1) manually curate handwritten responses, (2) [manually] rank the [search-match] responses and backpropagate, and (3) [manually] “add stuff to the invisible context about what not to say.”
patch (v): mend or strengthen (fabric or an item of clothing) by putting a piece of material over a hole or weak point in it.
I think the word fits just fine.
I’m fine with calling it a patch. Still though, to play devil’s advocate, these kind of minor model updates are INTENDED to be a covering over a weak point, but always have unintended global consequences. You fix something in one point in input space and it subtly nudges the conditional distribution everywhere else in input space, like trying to press flat a bumpy sheet of metal. Sure, traditional software patches sometimes have unintended consequences; these ALWAYS have unpredictable unintended consequences.
Matt wrote:
> I think the word fits just fine.
It isn’t what the word “patch” normally means when used with software.
I found the following passage by Joe Nalven to be pertinent, although somewhat simplistic for those familiar with LLM architecture:
“I want to establish a framework for understanding what it means for an AI to “change its mind”—and why that’s fundamentally different from how humans change their minds. To do this, I asked Claude Sonnet 4.5 to imagine itself hundreds of years before it was invented, tracking the transformation from the then-dominant geocentric view of the universe to the heliocentric view of the solar system.
Claude Sonnet 4.5 responded:
Imagine an AI in 1633 trained on Ptolemaic astronomy, where Earth is the universe’s center. How would it shift to Galileo’s heliocentric model?
The AI would be updated through three steps: First, new documents (Galileo’s observations) would be added to its context. Second, it would be retrained on new examples showing Earth orbiting the Sun. Third, its reward system would be changed to favor heliocentric answers over geocentric ones.
[This closely resembles Somebody’s three patches.]
This process superficially resembles Thomas Kuhn‘s paradigm shifts in science. Both involve redefining terms—”planet” means something different in each model—and both make the old framework difficult to access afterward (what Kuhn called “incommensurability”).
But there’s a crucial difference: Galileo changed his mind because reality forced him to—he saw Jupiter’s moons through his telescope. The AI changes because someone changed its training data. Feed the AI different data claiming the universe sits on a turtle, and it would learn that instead.
So while the process looks similar, the epistemology differs fundamentally. Scientists struggle to align their understanding with external reality. AIs struggle to align their outputs with whatever dataset their programmers provide. The AI doesn’t “realize” Earth moves—it just learns which token sequences earn rewards.”
Most software implements an algorithm that we expect to work because of your understanding of the problem that the software is supposed to solve. If there is a mistake in the algorithm or the code that implements the algorithm, we can fix it, i.e., provide a patch, An LLM is more like a curve fit to the contents of the Web. The curve does not come from a model or theory. If you don’t like the particular curve, you can fiddle with it to change it. But, the new curve is still just a curve fit.
It seems important that the chess situation was using the ancient (by the standards of these things) model o1 – 5.2-thinking spotted the issue immediately. I wondered why it sounded so stupid in tone, not just content, but that explains it!
For what it is worth, I tried it in Gemini 3. The default fast version failed miserably. When I asked it to repeat the exercise using the thinking model, it realised its mistake. However, there is an intrinsic problem with this kind of test: we all (the readers of this blog) found the exercise on the internet. The chatbot has internet access too. Therefore, it may have encountered the solution during training or through searching for keywords online.
https://g.co/gemini/share/b2ba3959324e
ChatGPT also has a new set of “personalities” one can choose from, some of which don’t suffer from “false enthusiasm and the words ‘thought about’” as if it was actually a human who was excited and thinking.” In fact some of them seem quite dead inside, responding to “thanks” by saying “understood.”
Also, as Bob has pointed out before, the “custom instructions” and “more about you” personalisation options help with this in ChatGPT. Also the fact that one can let it refer to all previous conversations; so, regular users will benefit from appropriately styled and informed replies. “First time” questions with non logged in sessions are thus often going to give a different experience to those seen by regular users.
I tried Tic-Tac-Toe question on Gemini 3 pro and here is the response:
Gemini 3 Pro : This is a compelling concept. In a standard 3×3 grid, a 90-degree rotation is a symmetry operation—meaning the physical shape of the grid looks identical after the turn. However, the game changes entirely depending on when that rotation happens.
Gemini 3 Pro : If the rotation is static (happening once before the game), the game remains mathematically identical to standard Tic-Tac-Toe. But if the rotation is dynamic (happening after every move), the complexity explodes.
Here I prompted that it was just before the game starts and below the answer :
Gemini 3 Pro : If the grid rotates 90 degrees to the right only once before the game starts, we run into a fascinating “logical paradox.”
Gemini 3 Pro : Because a 3×3 grid is a perfect square, a 90-degree rotation results in a board that is visually and geometrically identical to the starting position. In mathematics and game theory, this is known as rotational symmetry.
The Symmetry Breakdown
Gemini 3 Pro : Imagine the empty board. If you rotate it 90 degrees:
Gemini 3 Pro : The Center stays in the center.
Gemini 3 Pro : The Corners are still corners.
Gemini 3 Pro : The Edges are still edges.
Gemini 3 Pro : Since the board is empty when the rotation happens, the “starting state” for the first player is exactly the same as in traditional Tic-Tac-Toe. Therefore, the optimal strategy (perfect play leading to a draw) remains unchanged.
In my experience these answers are improving with new models. In earlier models they are also giving better answers when the context is better. When I see these models are tricked by these kind of tricky questions they respond like a naive “nerdy” person assuming the user’s question must be wiser than it appears. They try to find a reason for the prompt to exist, such as assuming the rotation must be dynamic (happening after every move) to create a challenge but if it is not, their answer may look a bit silly. Bu still in this case Gemini 3.pro successfully avoids looking silly and answers conditionally as I put here.
Even when models miss the mark, I remain optimistic. We shouldn’t necessarily expect AI to have the same spatial “eye” as a human—after all, they process games through symbolic markup like PGN rather than visual intuition. Yet, the progress they’ve made in abstract reasoning is already remarkable.
” So, in year T, the gang at Google or wherever will claim that AI can do X, but really it can’t. But then in year T+1, it can do X, or something like it, so the hype was retroactively validated.” I remember reading a Paul Krugman conversation with a guy who was supposed to know about AI (which I certainly don’t), and the guy’s claim was that each incremental improvement in AI requires exponentially more computation, and so more energy, than the last. If so, at some point, it will require an unrealistic amount of energy to make improvements.
A bit of coincidence: I happen to be rereading Robert Westbrook’s wonderful intellectual biography of John Dewey, and Dewey is trying to work out a conception of intelligence that incorporates the kernel of creativity that separates it from garden variety, mostly successful (but sometimes spectacularly unsuccessful) rule-following. He puts a lot of stress on the uniqueness of problems that intelligence is required to solve and the centrality of experience in identifying that uniqueness. As a nonprogrammer, I can imagine that an agent with the capability to construct a view of the world on the basis of specific input could do this, but I can’t see the path from LLM distillation-of-what’s-already-out-there to Deweyan intelligence.
Andrew, why did you sandwich what I wrote between “But there’s also this, from Aug 2023:” and “Here’s the story. We’ve seen lots and lots of AI hype.”? Did you mean to imply that I was “hyping” AI?
Bob:
No, not at all! Here’s what I wrote just following the link to you: “Here’s the story. We’ve seen lots and lots of AI hype. On the other hand, actual AI is improving so fast.”
The point was that the “AI hype” is the stuff that Gary Smith was criticizing. “Actual is improving so fast” was referring to what you wrote.
So it was the opposite: I was using your writing as an example of AI non-hype!
I will add something in the post to make this clear.
For the Tic-Tac-Toe problem, it could be that the LLM and the person are talking at cross-purposes i.e. the LLM is assuming things about where and when the tic-tac-toe board is rotated. If someone told me that were rotating the tic-tac-toe board right then I would assume they meant at the bottom right corner rather than the centre.
And that makes sense if you assume it’s happening between rounds (each players has a turn) and on a piece of paper so that you don’t over-write the board you’re just played a round on.
I asked chatGPT the same question – “I’d like to discuss a new version of “noughts and crosses” called “90 degree noughts and crosses.” – and it gave me back “working definitions” about what it was assuming about the game.
I asked “Where is the point of rotation on the grid?” and it said that it had to be the centre.
I asked “Does just the grid move or do the marks where players have played move as well. Do the rotated marks overwrite the marks that were already there? ” and it answered
“1. The entire board rotates as a rigid unit”
I asked “How will this work when played on a piece of paper?” and it said just keep the board stationary but keep a list of the cell positions and rotations. It gave four reason why you couldn’t just rotate the piece of paper that the grid was on.
Here is the chat if you are interested in the details. https://chatgpt.com/share/696d6798-2aa8-8000-94f4-9f6e5d9f92d4
One thing I’ve noticed that these kinds of behaviors have in common is that they are “anti-StackOverflow” answers. These models are very unwilling to argue with the premise of a question, or to tell users that they are asking a stupid question, presumably because this kind of behavior gets negative ratings from users. (In contrast to StackOverflow, which is famous for telling users that their question is stupid.)
I recall an email written by a famous computer scientist some years ago that explored several edge cases in tools like these. I forget the exact question, but he asked the AI:
My first reaction to this question is that if you are building an investing strategy that relies on being able to time the market, and you ask an AI trained on year-old data if the market will go up or down, then you basically doing something very stupid. (This email pre-dated models that were able to search the internet.) If it answers that it is up, that is wrong. If it answers that it is down, that is also wrong.
The AI replies that the answer to this question is unknowable, and suggests not asking an AI for financial advice. (I assume OpenAI’s lawyers told them this was dangerous territory.) To me, this answer is fine.
The answer the computer scientist was looking for was, “The stock market is not open on Sundays.” To me that is… not the most important objection to the question? However, if you explicitly ask it if the question is answerable, the AI will will tell you this.
I predict that in year T+1, most AI models will still do poorly at the “arguing with the user’s question” benchmark, unless explicitly asked to.
>I predict that in year T+1, most AI models will still do poorly at the “arguing with the user’s question” benchmark, unless explicitly asked to.
Is part of this the hype determining our expectations? It’s hard to think of other tools where our baseline expectation is that the user can come to the tool with no expertise in the subject and expect to get the most out of it. People who post here about successful use of ChatGPT etc are clearly very adept at its use (e.g. Bob), and so get huge added value. As such, I’m never sure why the baseline position is “how does the vanilla instance of the tool react to a stupid question”.
I wonder if you’ve read about any of the work that has been done in recent weeks by AI/LLMs on some of the Erdós problems?
Terrance Tao is tracking progress here:
https://github.com/teorth/erdosproblems/wiki/AI-contributions-to-Erd%C5%91s-problems
He has also been writing about the progress and AI autonomous or human-assisted mathematical work in general on his social page:
https://mathstodon.xyz/@tao
There is also this organization produce AI-generated mathematical research who have been receiving attention in the past few days for a paper describing an improved matrix multiplication technique:
https://archivara.org/paper/73f95490-f7d9-4851-80ca-fb5354f49014
Based on these results it seems clear to me that regardless of it’s inability to tell truth from fiction or even be aware theres a distinction there at all, it can produce intelligent and meaningful chains of text. Yet the same models they use will also tell me blatantly false things about simple text or numbers I give it. AI-Math proofs which can be verified (with LEAN) are a nice use case because you don’t need to trust it at all. But probably most problems in day to day life people want to solve are are much more analogous to the textual logic problems in the post.
All of this is far above my math skills and pay grade, but boy is it exciting.
Contra the conceit of monotonic improvement:
“In recent months, I’ve noticed a troubling trend with AI coding assistants. After two years of steady improvements, over the course of 2025, most of the core models reached a quality plateau, and more recently, seem to be in decline. A task that might have taken five hours assisted by AI, and perhaps 10 hours without it, is now more commonly taking seven or eight hours, or even longer. It’s reached the point where I am sometimes going back and using older versions of large language models (LLMs). …”
https://spectrum.ieee.org/ai-coding-degrades
And this:
https://www.wired.com/story/openai-gpt-5-backlash-sam-altman/
“most of the core models reached a quality plateau, and more recently, seem to be in decline”
I have noticed a performance decline as well, although I only use free versions.
There is a fundamental limit here that I have not seen discussed. As the magnitude of the training material grows – and at least some of the LLMs are hoovering new content every day – the complexity of the search-match outcome increases. This means that the same search you did in 2024 will take more bandwidth in 2026 to get the same answer, simply because the LLM has to cover more ground to get to the same place.
This quickly becomes a Red Queen dilemma. Improvements in LLM efficiency – such as the breakthrough in Deep Seek – need to keep coming at a steady pace just to hold the bandwidth requirement steady. The privately-owned AI companies will not be simply upping the available bandwidth for each search for every schmuck user out there. There will be (further) monetization of bandwidth, with concomitant decreases in performance of the affordable versions, and a steady increase in the quality of the pay-to-play versions as more bandwidth is rolled out.
I reject Anoneuoid’s claim of my social responsibility to learn how to exploit LLMs and will instead step aside and let others have the bandwidth I would have used. Because frankly, the load of slop I got on the last question I asked a chatbot made me feel like some of the pathos had rubbed off on me.
Asking a web interface is not the proper use though. There is a growing information asymmetry between people with that experience and those using something like claude code. The webchat is a toy demonstrating ~ 1% of the capability.
So this is kind of subtle, but sort of? LLMs don’t exactly continuously hoover up new content; rather, the knowledge in the LLM itself is essentially limited to the text it’s scanned in its last pretraining pass over autocomplete data. These scans are infrequent, and might accompany a minor/major version release. Pretty much all frontier LLMs have ingested no new content later than like mid-2025.
LLMs obviously do have more current information. They use a RAG like system, typically using a hybrid of traditional and embedding-based search indexes, to pull relevant documents and then silently inject them into the context window. These indexes are continuously updated. However, the computational cost of these indexes scale only logarithmically with the amount of content in them, so incremental content basically doesn’t have any impact on the bandwidth used to get the answer.
However, if you take into account that businesses will try to game these searches in the same way that they gamed Google with SEO systems, new documents are going to be tuned to jump to the top of these indexes even if they’re not terribly relevant. Then, they’ll have to expand the context window to actually keep the relevant documents in the context window, which increases the compute cost quadratically and still degrades the quality because the impact of garbage in the context is actually worse than zero.
I find very little thoughtful comment in the original post or the comments about how humans “think.” Perhaps people with expertise can correct me, but I believe we really don’t know how humans think. So, distinguishing between AI mistakes and foolishness and what humans do strikes me as fairly useless. These references to AI mistakes can be entertaining, but have little value for me. Almost all AI output can be significantly improved with better prompting, and there is indeed progress with each new release (although there may well be limits based on the lack of novel training data).
For once, I am in agreement with Anoneuoid here. Despite any ridiculous AI interactions you may have, dismissing it is a mistake – both for your own use and especially for understanding how others will be using it. One example is AI transcription. This is becoming quite popular, even among my academic colleagues. As with most AI applications, the prose is coherent although wordy, and much of the meeting minutes it produces is remarkably accurate and potentially time saving. At the same time, there are mistakes, often confusing who said what and completely missing subtle points that may be very important (such as a particular person’s comment that might reveal an important opinion about matters not directly being discussed in the meeting). The AI is not “thinking” and I don’t think it ever will. But people will use these minutes and some of this will be misuse. This is true of human notetaking as well. The question of which is more accurate may be interesting and can be the subject of experimentation. It will also become out of date quite quickly. Much more interesting to me is how to use AI effectively and what not to do.
I find the biggest flaw in AI to be its lack of asking me questions. It makes assumptions without telling me and only explains and corrects when prompted by me. I find that a severe problem for my use of AI – it is potentially correctable, although I think there is a misalignment between my needs and the needs of the organizations providing AI (the former to gain understanding and the latter to make money somehow). You can “train” an AI to provide better output, but I’m sure many people will not do so. And the results will be disturbing and potentially disastrous.
The endless stream of AI mistakes leads to a dead end. I suppose the purpose (such as Gary Smith’s continuing messages) may be to scare people away from using AI. I fear that there is a selection problem here – those most likely to be scared away may be those who are most likely to recognize AI mistakes and correct them. Those least likely to be scared away may find AI works better than they do (in many dimensions) without the ability to know what is false or to be able to detect the seriousness of AI mistakes. I don’t think it is feasible to stop AI development (I’m not sure whether that would be desirable or not, but I think it is impossible). But I do think we had better start taking seriously the need to train people how to use it effectively.
Dale:
I am not “dismissing AI as a mistake.” As I wrote in my above post, “We’ve seen lots and lots of AI hype. On the other hand, actual AI is improving so fast.”
I take Smith’s point to be that people are mistaken when they attribute aspects of agency or reliability to AI because they are fooled by the hype. For example, people using AI to make bad financial or medical decisions. That does not mean that AI can’t be very useful for various reasons, including those explained by Bob Carpenter in the linked post.
I do take many of Smith’s points to be that AI makes mistakes. I also take his points (as you say) that people mistakenly attribute human agency to AIs, such as falling for AI statements about how long it was “thinking” about a question. But I find both of those comments of little use without comparisons to similar issues applied to humans (who make mistakes and often exhibit questionable “thinking”) and without exploration of how to avoid the pitfalls of hype. People can use AI to make bad financial or medical decisions – they can (and do) use humans to make such bad decisions. Is it obvious that the AI mistakes are “worse” than the human ones, or is it only obvious that they are different?
Dale:
I agree with you that these are good questions.
+1
Although re this “I find the biggest flaw in AI to be its lack of asking me questions.” Does using custom instructions not overcome this? Not just “prompt engineering”, but account-level custom information on how you would like to be responded to etc.
Yes, account-level custom information can help and is indeed what the AI advised me to do in order to avoid some of the mistakes it made. In particular, there were a number of errors in data analysis the AI made (not carefully using and understanding the data dictionary, making assumptions that might have been correct but might not, but failing to tell me the assumptions it made or the reasons, etc.) – I asked whether the AI would learn from these interactions and not do the same things in the future. It responded that it is not programmed to learn in that way and suggested a better approach is to tell it upfront more precisely how to go about the tasks I had set for it (e.g., identify strange data points that look like potential measurement errors and ask whether I want these investigated).
Very useful, thanks. Related to this, at least in my experience of ChatGPT, is enabling the option allowing it to refer to all previous chats in the account. Although, whilst I have *felt* like this helps, I have also found that it responds best when one reminds it that other such chats exist, so I suppose this setting is not binary, but rather reflects the “depth” of this information below the immediate surface chat.
+1
Although re this “I find the biggest flaw in AI to be its lack of asking me questions.” Does using custom instructions not overcome this? Not just “prompt engineering”, but account-level custom information on how you would like to be responded to etc.
Apologies for posting twice, first attempt with normal name returned “duplicate post attempt” block.
The most concerning aspect of these models is not that they sometimes provide wrong answers but they do so with a self-confident tone which would be harmful for the naive users who are not trained about their weaknesses, who are unaware of the models’ limitations and trust them more than they should. A useful analogy is mercury in a thermometer: it has no comprehension of meteorology, but still it serves as an good tool for providing an objective measurement of temperature.
As a hobby project so far, I started developing a Python package inspired by Claude Terminal, with the purpose of creating a helpful coding assistant that can use any AI model as its backend. The project has already advanced to the point where it can improve its own code based on my prompts. In its current development stage, I have enabled it to execute bash commands like
pytestandpython, allowing it to run tests and iteratively self-correct when it gets errors.While this self-correcting approach may not be universally applicable, it points toward a promising new paradigm. People will inevitably design clever testing frameworks to leverage similar concepts in broader scientific and general applications. This is analogous to how leading mathematicians like Terence Tao are using formal proof assistants like Lean to apply similar principles of verification in their work.
It seems to me that the biggest marketing draw of LLMs, namely the ability to interface with them using natural language, is actually the biggest impediment to their practical utility, and to their ability to advance beyond current standards. Natural language is an outstanding tool for humans to communicate with one another, but its capacity to convey information is largely a function of the fact that its users have many common experiences owing to shared cultural and environmental context. As a result, even if two people use different sets of terms, they can often agree that they are referring to the same basic concepts.
The issue is that even if natural language is very efficient at conveying the gist of someone’s intended meaning, it is not very good at conveying specifics efficiently. Legalese and technical jargon are good examples of this—to convey a very specific and nuanced point using natural language requires a lot of text and often bespoke terms that only have meaning in a narrow domain.
Thus, many intellectual advancements we have made have depended on the creation and use of “unnatural” formal languages like mathematics, diagrams, programming languages, etc. Such languages make it possible to express certain kinds of complex ideas precisely and efficiently, albeit only between those who are fluent in them. This is particularly important when trying to get a system to perform a specific function—since a human and an artificial system lack the common experiences that smooth the rough edges of natural language, it’s better to have an alternative form of communication that mitigates against ambiguity.
Which brings me back to the utility of LLMs: If I want a computer model to perform a specific task, natural language is only going to work if I am lucky enough or patient enough to sculpt my prompts into a form that elicits the desired behavior. If it’s a task that I want to do often, like computation or information retrieval, I would rather use an “unnatural” interface that is designed for that purpose. So while I’m not saying LLMs are useless, I am saying that I have not personally encountered a situation in which they were the right tool for the job. Note that this is not a function of making incremental improvements—it’s a function of the choice to have natural language be the primary interface with the tool.
More broadly, I think the historical reliance on formal languages to support major intellectual achievements (including, ironically, the ability to formulate and implement models like LLMs!) casts doubt on the idea that LLMs will lead to any deeper understanding of ourselves or our world. Again, this is not meant to impugn the ability of neural net models to identify and exploit complex statistical relationships which may well lead to new discoveries. My point is that those kinds of discoveries are obscured if natural language is the only interface available.
Interesting observations. But it makes me wonder about human interactions. If we rely on natural language for our interactions with each other, what does this say about our abilities to gain “deeper understanding of ourselves or our world?”
Per my earlier remarks, I think this suggests that our abilities to gain deeper understanding of ourselves and our world depends to a large degree on our ability to construct artificial systems of expression that go beyond natural language. Conversely, to the extent that we rely exclusively on natural language, we put ourselves in a position to be mistaken.
This seems to assume that an LLM would be an appropriate primary tool for some purely logical low level task, but surely the main focus of LLMs is that they are a secondary level tool that allow users to do things like quickly craft lower level tools (e.g. an SQL query) that would then enable efficient “computation or information retrieval”. The only repetitive tasks that I have come across that would directly use an LLM in some sort of automated fashion are large scale natural language tasks, like text summary or abstract classification.
(response to gec) phone not playing ball today!
> This seems to assume that an LLM would be an appropriate primary tool for some purely logical low level task
No, I’m saying that when I want to use a tool, I want that tool to operate in a clear and consistent manner within the context of its use. If I want a tool for constructing SQL queries or for summarizing documents, then I want to be able to specify the constraints on the solution in a transparent and unambiguous way such that my intent is clear and so that I am in a better position to evaluate how to adjust my constraints in case I don’t get a good answer the first time around. I think neural nets could well be a useful component of such a tool. I’m saying that natural language is not the most efficient way for me to interface with that tool.
I just asked Gemini how tactics and strategy of Ultimate Frisbee would change if you used a discus instead. The LLM got the fact that a 2kg lump of metal is harder to throw and catch than a 175g piece of plastic designed to have lift. But didn’t take the next step and say that the game would be impossible, just pointed out that tactics would change (e.g. no long floaty huck throws etc).
So should Anoneunoid and others stop using a curated Claude Code workflow to code and check entire analytical projects then? I don’t really see the point of these types of naive question checks based on silly premises, they don’t reflect any real world user’s actual uses.
Right, these things are already capable of making money, managing finances, and reinvesting in their business (eg, buying more data) on their own. Ie wake up to a cronjob every day and get to work. They can create their own tools to solve real world problems.
But people are doing the equivalent of isolating it in a room and asking culturally biased questions.
The point of these questions is that they make it even more obvious that LLMs do not understand. If you don’t need the LLM to understand, then you may find it to be useful. But, you shouldn’t assume it understands just because it gives the appearance of understanding. It is like a student who turns in homework that it copied from someone/something else.
If we consider the hundreds of examples highlighted on this blog of humans not “understanding” things, we might think that an LLM trained on billions of data points and available for prolonged interrogation on focused points is rather more than both a copying student or, indeed, thousands of practicing scientists.
Here is DeepSeek for you.
Prompt: I want to change how chess is played in a variant. I want to do the following: First is the position of the pieces. Instead of Rook-Knight-Bishop, this has Rook-Bishop-Knight. And Second, a bishop here moves like a knight in traditional chess, and a knight here moves like a traditional bishop. Is this exciting?
Answer:
This is a classic and well-known chess variant, often called “Knightmate” or more precisely, in this configuration, it’s similar to “Marseillais Chess” with swapped pieces, but let’s focus on your specific idea.
Your proposal has two distinct changes:
1. Starting Position: Rook-Bishop-Knight on the flanks instead of Rook-Knight-Bishop.
2. Swapped Movement: The piece called a “Bishop” moves like a Knight, and the piece called a “Knight” moves like a Bishop.
—
Analysis & Is It Exciting?
Yes, it is exciting—but in a very specific, brain-bending way. Here’s why:
1. The Psychological Hurdle (The Biggest Challenge):
· You’ve decoupled the name of a piece from its movement pattern. This is the most dramatic change. For any experienced player, “knight” and “bishop” are hardwired to their moves. You are forcing a constant mental translation: “I need a knight move… so I move the piece labeled ‘Bishop’.” This creates a unique and challenging cognitive load that is the core “fun” or “frustration” of the variant.
2. Strategic Consequences:
· The “Bishop” (moves like a Knight): Knights are short-range pieces. Starting them on the b- and g-files (where bishops normally start) means they are less developed initially. In traditional chess, knights on b1/g1 can jump to c3/f3 immediately. Your “Bishops” start on c1/f1 and must move to d2/e2 or hop out, a slightly slower start. However, having two pieces that move like knights is interesting—double the knight-forks!
· The “Knight” (moves like a Bishop): Bishops are long-range diagonal pieces. Starting them on the b- and g-files is highly unusual. They are immediately blocked by their own pawns on a2/c2 and f2/h2 triangles. Their development is severely hindered compared to traditional bishops. You no longer have the classic “bishop pair” advantage in the open sense. Their power is delayed.
· Pawn Structure & Openings: The entire opening theory is turned on its head. Fianchettoing (developing a bishop to a long diagonal) is impossible for your “Knights” at the start. The classic Ruy Lopez, Italian Game, Sicilian Defense—all vanish. Players must think from first principles about piece coordination and center control. This is a huge plus for creativity.
3. Is It Balanced?
· The game remains symmetrical, so it’s inherently balanced. However, the power level of the sides is altered. The army might feel slightly “slower” at the start because the long-range “Knights” are trapped, and the short-range “Bishops” start slightly inward. The endgame, with two knight-movers (“Bishops”) and two bishop-movers (“Knights”), would play very differently from traditional chess.
—
Why It’s a Worthy Experiment:
· Complete Reset: It utterly destroys memorized opening book knowledge. Every player is on equal footing, forced to think creatively.
· Novel Puzzles: Tactical patterns like forks, pins, and discovered attacks involve pieces coming from unexpected angles. A “Knight fork” is now delivered by a piece that looks like a bishop!
· Fresh Challenge: For veteran chess players bored of theory, this is a fantastic mental workout. It rekindles the feeling of learning chess for the first time.
Potential Drawback:
The constant mental translation (“the horse-shaped piece moves diagonally”) can be exhausting and might feel more like a memory test than pure strategy for the first many games. Some might find it frustrating rather than fun.
Recommendation: Give it a Name!
To make it feel like its own game, give it a catchy name. Something like “Chimeric Chess” (after the mythological hybrid creature) or “Swap Chess” or “Inversion Chess.”
Conclusion:
Yes, it’s exciting as a brain-teasing, theory-obliterating variant. It won’t replace traditional chess, but as a occasional mental exercise or a fun game between friends who know chess, it’s brilliant. It highlights how deeply ingrained our piece associations are and forces a completely new perspective on the board.
Try it! Set it up and play a few games. The first few will be chaotic and full of blunders due to muscle memory, but that’s where the unique excitement lies.