Before going on, let me emphasize that, yes, modern AI is absolutely amazing—self-driving cars, machines that can play ping-pong, chessbots, computer programs that write sonnets, the whole deal! Call it machine intelligence or whatever, it’s amazing.
What I’m getting at in this post is the way in which attitudes toward AI fit into existing practices in science and other aspects of life.
This came up recently in comments:
“AI” does not just refer to a particular set of algorithms or computer programs but also to the attitude in which an algorithm or computer program is idealized to the extent that people think it’s ok for them to rely on it and not engage their brains.
Some examples of “AI” in that sense of the term:
– When people put a car on self-driving mode and then disengage from the wheel.
– When people send out a memo produced by a chatbot without reading and understanding it first.
– When researchers use regression discontinuity analysis or some other identification strategy and don’t check that their numbers make any sense at all.
– When journal editors see outrageous claims backed “p less than 0.05” and then just push the Publish button.“AI” is all around us, if you just know where to look!
One thing that interests me here is how current expectations of AI in some ways match and in some ways go beyond past conceptions in science fiction. The chatbot, for example, is pretty similar to all those talking robots, and I guess you could imagine a kid in such a story asking his robot to do his homework for him. Maybe the difference is that the robot is thought to have some sort of autonomy, along which comes some idiosyncratic fallibility (if only that the robot is too “logical” to always see clearly to the solution of a problem), whereas an AI is considered more of an institutional product with some sort of reliability, in the same sense that every bottle of Coca-Cola is the same. Maybe that’s the connection to naive trust in standardized statistical methods.
This also relates to the idea that humans used to be thought of as the rational animal but now are viewed as irrational computers. In the past, our rationality was considered to be what separates us from the beasts, either individually or through collective action, as in Locke and Hobbes. If the comparison point is animals, then our rationality is a real plus! Nowadays, though, it seems almost the opposite: if the comparison point is a computer, then what makes us special is not our rationality but our emotions.
I like this approach – it goes beyond the usual (and what I find superficial) issues of whether AI “understands” to what the real dangers of AI are. If I think of the psychological constructs of systems 1 and 2, the blind and misplaced trust in AI seems like system 1 thinking. Our minds are lazy (without the pejorative connotation) and AI saves us from having to think. Of course, that is dangerous, although useful and even necessary at times – the complexity of technology makes unconscious reliance on technology impossible (and unwise) to resist at times. But we, as a species, don’t seem to have evolved the ability to know when to rely on technology and when not to.
Here’s an example: I often rely on Google searches. The more and better AI that is built into search, the more sensible it is to rely on it. However, the commercial interests of Google have increasingly made this dangerous and unreliable. I have to engage system 2 to override the easy reliance on system 1 to search the internet for information I desire. This requires effort on my part, but is necessary at this point. Improvements in AI might save me that effort someday, but not unless the incentive structure is changed so that I can afford to rely on search without engaging system 2 effort.
Where does rationality fit in? I’m not sure that it does. It is perfectly rational to “trust” AI under certain circumstances but not under others. Human emotions certainly distinguish us from computers, but I don’t think the dimension to measure is accuracy, efficiency, or rationality. Emotions can only get in the way of these things. It is not emotion that protects us against dangerous AI – it is actually critical thought and rationality that inform us when we should not trust AI. Emotion can alert us to dangers, but just as easily it can mislead us to see danger where it is not.
What distinguishes humans from computers I see as “responsibility” and “accountability.” I have joked that I want to protect the right of humans to make mistakes. Even if an AI can do some things better than humans, I think it is important that we be held accountable for our decisions. It is hard to attribute responsibility to an AI – who exactly is the creator of an AI? (I think I’ve wandered a bit in this comment, but the post raises many different issues in my mind).
Hey! You called???
Your point is excellent, spot on, well taken. The phenomenon you are talking about is a repetition of the 1950s and 60s idea of the computer as an “electronic brain” and companies using “the computer did it” as an excuse. But it’s a more general thing, not just AI: it’s the problem of placing excessive trust in any black box.
I’m more concerned with a subtler problem: a lot of scientific papers, even in Science, seem to be of the form “We used AI to do X”, but what they were actually doing was some sort of calculation. Calculation is the purview of the non-AI parts of Computer Science (and math and statistics), and I suspect that some of those calculations have proerties that aren’t well explicated yet, and really aren’t reliable. And that the authors don’t fully understand what their calclulations actually computed.
This probem is especially bad when combined with one of my favorite anti-AI rants (yes, I can hear you groaning) namely that neural nets don’t look anything like actual neurons (and, by the way, that you can use a large neural net to simulate a single actual neuron (but not the connectivity, the very thing the neural net folks are claiming for neural nets!) does NOT show that neural nets are about neurons, only that they are something like Turing equivalent and can be used to perform arbitrary calculations.). Which means that the vast numbers of folks using neural nets and claiming their results have validity because they’re “like the brain” is all dead wrong.
Whatever. I remain completely surprised that large numbers of otherwise apparently sensible human beings aren’t disgusted by the mansplaining-like error-riddled text spewed out by ChatGPT and the like. Goggle seems that it’s going to be trying to charge me for a service I’d pay to avoid. Seems pretty hilarious. Here, so far, Google remains better than Microsoft for search. Bing is terrible about only showing commercial sites. With Google, I just append “wiki” to my search string and get something useful almost every time… (I can find my photo page by googling for “pbase” and “davidj”, but it doesn’t work with other search engines.)
And to leave you with the latest trully depressing claim from the anti-AI space: the AI types claim that vast increases in the amount of training data are associated with at least some improvement in “performance”. But that assumes _QUALITY_ training data. And if you train your LLM on other LLM’s output (which the internet is said to be getting full of), your performance goes down. Rapidly.
You forgot to mention “parlor games.” I actually agree with many of your points (particularly about the ill-conceived description of neural networks), but not your major point (I think). I have a drone – it is fairly amazing technology and I don’t understand at all how it does what it does. But I enjoy it nonetheless and don’t see a problem with that (unless, of course, it were a war-equipped drone). Further, you might take a look at this week’s Economist with their feature on AI in medicine. Sure there’s much hype, but also much to be impressed by. Computers do what they do very well – the mistake is to think they do something different that they are not at all good at. Computers cannot be held accountable in any meaningful sense. But for every example you can give of how they don’t compute well, there are examples where they do. After all, that is what they are designed to do.
As for whether i should be “disgusted by the mansplaining-like error-riddled text spewed out by ChatGPT” I am not disgusted at all by that. Amused somewhat. Also, I know that it is relatively easy to work around those errors provided I let my brain continue to work. But the mansplaining-like error-riddled spewing that disgusts me is done by humans, such as he who will not be named.
Hey! It’s “parlor trick”, not “parlor game”. Don’t mess up my pejoratives! Parlor games are fine, parlor tricks aren’t.
Using ChatGPT for entertainment, or using it to access it’s collection of code templates that you can quickly fix is fine.
But the folks claiming that ChatGPT is useful in situations were (a) the answer is important and (b) the user needs a correct answer (that is, said user isn’t capable of recognizing an incorrect answer) are pure evil liars. ChatGPT can’t be used in that situation. But the hucksters who are selling ChatGPT won’t tell you that. (Again, if the user is a smart, educated, well-off academic (there are good reasons folks in this demographic have longer life-expectancies than those in certain other demographics), they can figure out that there’s a problem. But the “using AI in medicine” types leave that part out.)
By the way, I find it hard to believe you don’t understand the issues of feedback control required to keep a drone stable. Did you miss the Markov Chain Shakespeare generators, which are the intellectual grandfathers of the LLMs? If so, check them out. They were terrible, but occassionally generated something vaguely Shakespeare-esque. (Hint: “sequences of undefined tokens”. That’s all there is under the hood.)
FWIW, there’s a “Numberphile” YouTube video that explains GPT-2 quite nicely. I found it quite hilarious because the bloke describing it was hyper-enthused about said inanely trivial sequences of undefined tokens based parlor trick.
I probably wouldn’t pay for ChatGPT for it’s mansplaining error ridden qualities. But RAG systems that cite their sources are extremely helpful now, and I encourage you to check them out. Of course, they kind of lose some of the magical qualities of “AI”; RAG systems are pretty transparently a MIPS index paired with a learned syntax/grammar generator to stitch things together.
I think this is a sensible viewpoint. In general, I think that technology is only worthwhile to the extent that it extends human capabilities. As I say to my students, computers save us from having to do a lot of “mindless” tasks (like tedious computation), but then it falls on us to use the time saved to do more *mindful* work. The examples you give are all cases where technology accomplishes a “mindless” task but we don’t bother to back-fill that time with mindful work.
Like David, I don’t really see much value in many modern AI applications like Chatbots because I don’t see how they are enabling me to do any more mindful work. For example, if I want to use the output of a chatbot mindfully, I need to have expertise in the domain I’ve asked about so I can critically evaluate its output. But if I already have that expertise, what effort has been saved by using the chatbot? I might as well have just written it myself in less time.
That said, I’ve been very impressed with using modern AI tools for coding, as Bob Carpenter has described here. Writing code that works and is efficient is time-consuming. But a lot of coding amounts to re-implementing some basic structure so that it applies to a new case. For example, I may want to write Stan code to apply a particular type of model to data with a unique structure. If an AI model can get me that code in a few minutes, then I can spend more time exploring different models and coming to a better understanding of what is going on in my data. Moreover, testing code is a much better constrained problem than trying to critically evaluate a large chunk of loosely structured text, so even if expertise is needed to assess AI-produced code, that expertise can be deployed more efficiently. Thus, I view AI-for-code as an application that enables mindful work.
I think the fair way to evaluate ChatGPT is to consider what a human brain, with its wondrously-connected neurons, but grown in a vat, then forced to analyze a trillion ASCII codes from Internet sources for correlations, would be able to do with the results.
I wouldn’t expect its responses to ASCII prompts to be perfect or even good, so I am somewhat impressed by ChatGPT (the free 3.5 version), while not considering it much more than an early demonstration of future possibilities. Meanwhile, there are several known useful applications of neural-network AI, which I will forebear from listing yet again. (Although I have heard of some new ones recently, but anyone can find them who looks.)
I think a more fair way to evaluate something like ChatGPT would be to compare it to a different statistical model that does not carry the baggage of the word “neuron”.
For example, you could estimate a giant logistic regression model that uses the previous N words as predictors for the next word. I suspect that, with a large enough dataset and with large enough N, you could get a logistic regression model to produce sensible-looking output that is impressive in a similar way as ChatGPT. Churning through all that data would be a challenge, but I’m not sure it would be any less challenging than the computations needed to train ChatGPT currently, so it could be done.
The purpose would be to see how much of the “power” of these models comes from their neurally-inspired architectures (which, of course, have been around since at least the early 1980’s, arguably earlier) versus how much comes from the ability to crank out massive amounts of computation in a reasonable time.
Now let N vary and choose the best N for each case. It would be interesting to try, but I suspect it would be a big job, with the possibility of convergence problems.
The basic idea of neural networks, as I see it, is automated trial and error. Which appeals to me because I consider trial and error to be the fundamental basis of all progress; and how my neurons learn new skills, e.g., how to play guitar, and shoot baskets.
The form of trial and error used by neural networks, although it has been refined in many ways, may not be the best way to handle some problems, but it has proved useful in several applications so far.
“I consider trial and error to be the fundamental basis of all progress”
Well, a lot of progress, but that’s mostly low level progress. It doesn’t answer the question “what to try”. That’s the question wherein lies actual human brilliance.
Take guitar. So you want to play bebop like Grant Green or Sonny Stitt. It’s real nice (and completely necessary) to have a large repertoir of patterns and tricks for playing over specific chords and short sequences of chords, and getting those patterns and tricks under your fingers is repetition/trial and error learning. But how to string these things together to make music, and how to visualize the music as it’s going by in real time, that sort of stuff is much higher level.
In baseball, lots of hitters lift their towards-the-pitcher foot off the ground to help them get more power behind the swing. But one of Otani’s advisors pointed out that that was making him too slow for really fast fast balls. So Otani keeps his forward foot on the ground now, and has a higher batting average, especially against fast ball pitchers. Thinking really is useful some times.
> The basic idea of neural networks, as I see it, is automated trial and error.
I think this is technically correct, but I do not think it uniquely picks out neural nets as a class of model.
Neural networks work by iteratively adjusting a set of weights in order to minimize an error criterion. The thing that made neural nets viable in the early 1980’s was the development of “backpropagation” of that error signal, so that you could jointly learn multiple layers of weights.
Generally speaking, the complexity of neural nets means that these parameter adjustments must be done iteratively, which can be described as “automated trial and error”. However, any estimator that reduces error also has this property. Least squares regression could be accomplished by iteratively adjusting regression weights to minimize the squared error between predicted and observed. But, unlike a neural net, least squares regression is simple enough that we can take an analytic shortcut around that iteration.
In summary, all statistical models “learn” via trial and error, the difference is that some models are complex enough that you have to perform trial-and-error iteratively (like with a neural net) but other models are simple enough that you can “skip to the end” (like with least-squares regression).
I agree that there are other forms of trial and error than neural networks, as I suggested in my last paragraph.
The Otani example could be summed up as Otani tried one method, got some poor results, then tried another.
I should have said however that the full evolutionary algorithm was fundamental, not just the trial and error part: variation of trials (including but not limited to random), selection criteria, and memory.
My favorite example, currently, is Einstein trying a couple different methods of formulating General Relativity mathematically, which didn’t work, then hearing about the Riemann Equations from someone else’s memory.
Just to share this: https://www.ams.org/journals/bull/2024-61-02/.
BULLETIN OF THE AMERICAN MATHEMATICAL SOCIETY (Vol. 61, No. 2 April 2024)
Alejandro Adem, A note from the chief editor . . . . . . . . . . . . . . . . . . . . . . . . . . 199
Maia Fraser, Andrew Granville, Michael H. Harris, Colin McLarty,
Emily Riehl, and Akshay Venkatesh, Will machines change
mathematics? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Akshay Venkatesh, Some thoughts on automation and mathematical
research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Kevin Buzzard, Mathematical reasoning and the computer . . . . . . . . . . . . . . 211
Jeremy Avigad, Mathematics and the formal turn . . . . . . . . . . . . . . . . . . . . . . . 225
Johan Commelin and Adam Topaz, Abstraction boundaries and spec
driven development in pure mathematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Michael Shulman, Strange new universes: Proof assistants and synthetic
foundations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
Geordie Williamson, Is deep learning a useful tool for the pure
mathematician? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
Ernest Davis, Mathematics, word problems, common sense, and artificial
intelligence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
Eugenia Cheng, How machines can make mathematics more congressive . 305
Andrew Granville, Proof in the time of machines . . . . . . . . . . . . . . . . . . . . . . . 317
Michael Harris, Automation compels mathematicians to reflect on our
values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
Thanks for the link to this AMS Special Issue! Some big names and big interesting ideas. Thank you for ruining my weekend plans :)
(This is one of the very few blogs where scrolling through the comments make it worth your while)
Michael Harris at Silicon Reckoner is interesting.
https://siliconreckoner.substack.com/
As someone once on the Macsyma payroll, I don’t have any problem with formalizing mathematics, but from reading the very introductory things on the Langlands Program and the FLT proof, I think Harris has exactly the right idea on Mathematics as a _human_ discipline. That is, proofs are interesting when they involve new ways of looking at the problem, and a new proof (even of an already proven theorem) will give the field new directions to go in, and that’s something that’s not going to get automated.
You wouldn’t be a friend of my favorite author, Minato Kanae, would you???
I’m afraid not, David. I’m from Osaka and live in Virginia…
AI is really not new. Examples of existing AI misuse are:
1. Use the spellchecker and then don’t proofread to look for outrageous errors.
2. Use the finish-word feature in texting and don’t look to see how humorous it’s made your text.
3. Use Google to find a fact, and just quote the first entry to appear.
ChatGPT and such are sophisticated spell-checkers. And as with Google, it may be that the new technology ends up helping bright people more than dull people because they offer more opportunities to go wrong if you rely on them blindly.
Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them. -Frank Herbert
Dan Davies writes about “accountability sinks”
(P < 0.05 is the ultimate accountability sink)
https://backofmind.substack.com/p/elephants-fight-mice-get-flattened
The,
I’m a fan of Davies but I don’t think that particular post makes a lot of sense. He statement, “The overpowering political theme of my adult life has been a retreat, on the part of those with power, from any idea of making a decision and living with it,” seems like a classic “end of history” bit, but . . . those in power make all sorts of decisions and live with them. The Russian government invaded Ukraine, for example. That was a major decision that they took.
I feel like what Davies is saying is that when his preferred political parties are in power they don’t make decisions that he wants them to take. But he’s embedding that specific statement into an incorrect general statement.
Little late to the party, but excellent post. AI/ML/statistics don’t have judgement like we do. That’s why it’s important for people to combine information you get from models, as well as information you get from your best judgement.