Forking paths in LLMs for data analysis

Posted on by

This is Jessica. I spent last week at a workshop where we were asked to prepare a short provocation related to interactive data analysis (same workshop Andrew mentioned already).  In thinking about what could be said about the future of data analysis on the way there, I decided one can’t really consider the future of data analysis, including how to address issues of forking paths and replicability, without considering LLMs. 

This seemed like the right direction given that they asked for a provocation. After all, what better way to put people on edge these days than the cliche and annoying move of changing the topic of what is meant to be a serious academic conversation to focus on LLMs? 

But I was also being sincere. After I started thinking about it, the thought of spending a week thinking and talking about the future of interactive data analysis without engaging at all with language models seemed irresponsible in light of the current moment. 

For one, LLMs are already being used for scientific purposes beyond just writing and coding suggestions. Summarization and qualitative analysis of text corpora, simulating outcomes of large social systems, generating digital twins for individuals to estimate individual-level counterfactuals in medical trials, and so on. And so naturally researchers are exploring their use as general purpose tools for data-driven analysis and visualization. Wrapper systems that take a high level analysis goal like “analyze last year’s sales data” and interface with an LLM (and access to other statistical tools, e.g., for fitting the models) for the user are being developed. 

And some people are already doing exploratory forms of data analysis with them. For instance, OpenAI has a page on their site dedicated to their expanding features for ChatGPT-supported data analysis, with quotes like this:

ChatGPT is part of my toolkit for analyzing customer data, which has become too large and complex for Excel. It helps me sift through massive datasets, allowing me to conduct more data exploration on my own and reduce the time it takes to reach valuable insights.

Or this:

ChatGPT walks me through data analysis and helps me better understand insights. It makes my job more fulfilling, helps me learn, and frees up my time to focus on more strategic parts of my job.

To many people who care a lot about data analysis this may seem horrifying. But increasing the level of automation in data analysis workflows has been happening for years. It’s often our goal in our own personal practice to automate the tedious steps of modeling workflow and identify patterns and protocols to guide us through the harder decisions. So I find it interesting to reflect on why we might react especially negatively to the idea of incorporating greater levels of assistance and automation via LLMs. 

As a thinking prompt, imagine yourself before you knew much about stats. You have some data to analyze and can get help from either:

  1. A rule-driven assistant that has you answer some multiple choice questions about your goals and data spec and then recommends a modeling approach or test
  2. A (human) statistics consultant with an advanced stats degree who has a conversation with you then makes recommendations and is available to provide review as you complete your analysis.
  3. An LLM that attempts to perform the same role as the human consultant. 

What makes these options seem different, and do any of these differences lead us to believe one is much better than the others? 

One difference would seem to be that two of these things are blackboxes, which we might expect to be more unpredictable in output than the other. It seems easier to evaluate how #1 could fail. Its specification is more concise and interpretable. We would expect #2 and #3 to produce output for a much wider range of inputs, and the outputs might vary a bit depending on how you approach them, when you ask them, etc., making evaluation harder. Comparing the human consultant to the LLM, we would expect the LLM to have ingested a lot more examples, good and bad, and a lot more statistics texts. There’s potentially a huge amount of signal and a huge amount of noise in all this, and we don’t understand fully how it’s getting combined. The human’s training data and synthesis process is also hard to delineate. But maybe still more debuggable in that we might expect them to respond based on some smaller set of principles or protocols or heuristics that they refer to to organize their behavior. 

There are many other dimensions we could compare them along – learning rate, awareness of conventions across different domains, etc. Are any of the dimensions on which they vary enough to make one of them seem necessarily better? Do we believe that whatever level of machine assistance we are using now is the optimal level, and that an LLM couldn’t improve upon it? I don’t necessarily have answers or even strong opinions, but I find it interesting to think about.

I suspect some of the resistance to LLMs as analysis assistants stems from fears of further ratcheting up mindlessness in data analysis. We know that regardless of our noble attempts as researchers to educate on what good modeling practice looks like, many people seem to prefer a ritualized approach where statistical analysis is a black box that takes in data and spits out answers. LLMs are threatening because they would seem to make this kind of approach easier. For example, dealing with data formatting issues remains a major time-sink in analysis, especially for users who are not programmers, but ChatGPT at least superficially seems tolerant of whatever you want to paste in. This isn’t to say it won’t err to different extents with different formats, prompts etc., just that the interface is more opaque, which can give a sense of troubles being alleviated.

There’s also fear of “lowest common denominator” analyses, where the user pastes in their data and question and the model recommends or runs whatever is most popular. And of “poisoning the well” when people publish AI-assisted analysis results that exemplify this “data analysis for the masses” vibe which get pumped back into the model as training data.

Compared to code suggestions, which is where they are most likely to be used in modeling workflows today, looking to them for data analysis seems closer to letting them tell us what to believe (since learning from data is often about updating our beliefs). 

I wonder though how much the hesitance to consider AI-driven data analysis is the knee-jerk reaction that somehow human domain knowledge needs to be unrestricted and its application left open-ended, that anything iterative and interactive must be left to human control. As if even if we can better attain our stated scientific goals with AI, it will ultimately somehow still be worse.  I sense this attitude in fields like human-computer interaction and data visualization, where people sometimes resist the thought of formalizing the goals of some human-computer interaction, as if that automatically equates to taking agency from the human. When it comes to topics like visual data analysis it’s like we would prefer to just trust that the human knows what they are doing and will apply their domain knowledge in the most appropriate ways. 

This ignores that when we leave things fully to the human at the highest level, that is also a design choice, one that usually makes problems harder to find because we’re less clear on what we’re looking for. So I tend to think that when the thought of formalizing or automating some task makes us prickly, it’s probably something we could learn from thinking more about. If nothing else, the exercise of trying to imagine a data analysis assistant that optimally pairs human domain knowledge with AI-driven suggestions forces us to reflect a little harder on what exactly we think the human is doing/adding at each step, which is why I find it generative for thinking about the future of data analysis tools.  

I recall Andrew saying something during the workshop along the lines of ‘When it comes to data analysis we all like to think that our particular combination of flexibility and rigor is somehow the right balance.’ Which implies that we don’t think it’s obvious how to combine domain knowledge with methods, and we don’t necessarily trust others to figure it out. Why would we want to add another agent we don’t fully understand into the mix? Perhaps it’s natural to hesitate a bit when we sense boundaries in our knowledge, because we don’t want to make mistakes.

Ultimately, I’m not advocating for everyone working on data analysis tools to start thinking about LLMs. But I also don’t think we should let our knee-jerk resistance keep us from considering the bigger picture. Presumably there are also many ways in which they could help human analysts overcome limitations. For example, lately I’ve been thinking about how a paradox of learning from data is that you can’t do it without a good imagination. Many of the human errors we point out in data analysis can be attributed to a lack of ability to entertain multiple possibilities. We like to suppress and reduce uncertainty, not maintain it as we go. We don’t do sensitivity analyses as much as we should, nor take the results as seriously as we should. We don’t engage deeply with many of the assumptions behind standard choices. We regularly ignore forms of multiplicity, from the fact that the same ATE can be consistent with very different patterns of heterogeneity at the individual level to the way a machine learning pipeline can return a set of seemingly equivalently performing models. 

Part of the problem is that analysis is cognitive demanding, and there’s only so much we can keep in mind at once. Remaining aware of how different assumptions made along the way subtly impact the interpretation of results might be too high a bar for a person. Maybe designing LLM-based assistants with the express goal of helping us keep track of and critique assumptions, uncertainties, multiplicity, etc., and interrupt us with them at the right times could help curb overinterpretation of results.

It could also be interesting to think about how multiple agents who perform different roles (and get different access to different parts of the problem, including the data) could be combined to get around the various problems of leakage or data conditioning that we see in practice. The challenge is not boxing ourselves in by assuming overly strict designs, but that’s been true for all attempts to integrate more automation in data analysis.

Of course, simulating nuanced use of imagination to drive analysis will require learning what it looks like from somewhere. Ultimately factors like the availability of good prior analyses and our lack of sufficient control in how models are fine-tuned may be major challenges. I’m curious what else.

P.S. It occurs to me this post is partly about how different people react when they see something in their domain of expertise that could be a train wreck starting to occur. Do you jump in and try to redirect it, or avoid the situation altogether? I guess in this case I’m advocating that at least some of the experts jump in.

169 thoughts on “Forking paths in LLMs for data analysis”

  1. Much as I think LLMs rival human abilities for many things, I admit that I am reluctant to use them for my own data analysis. I’ve resisted simpler automated tools in the past – for data visualization, I abhor the software “recommendations” that many software products provide for such things. I find that every data set is unique and I also know my own knowledge is severely limited. So, it is a matter of deciding the best way to proceed: invest energy in exploring the data more fully, learning more about methods that I don’t know well, asking experts for help, reading related publications to gain new insights, or seeking assistance from an LLM. Among the choices, I think the returns from using the LLM are the lowest of these. That may not be true for everyone. If I knew less (is that possible?), maybe the cost/benefit calculation would be different. If I did coding, then perhaps I would find LLMs a time saver. But for the type of analysis I do, and the type of teaching I do, I just don’t see the LLMs as being a good use of my time.

    Perhaps I will need to concede to thinking as a number of others on this blog have repeatedly maintained. I know the LLMs cannot “think” so I don’t see how they can help with unraveling the stories in the data. If I have a specific task to do (e.g., how do I calculate a Brier score?), I can and do use automated tools. But the more meaningful questions: what is actually being measured and how, is the data accurate, is there some transformation that will fit the data in an interesting way, what is the best recoding for a complex nominal variable, etc. I just don’t trust the LLM’s training data to be rich enough to make it worth my time. I don’t think it has been trained to do these things – it has been trained to find patterns from its training data for what I am asking, and I thing the likelihood of getting much of value is not worth the opportunity cost.

    Reply ↓

    • I can relate… I haven’t really had the patience to figure out how to make them helpful for most of the tasks I do routinely. But I find data analysis with LLMs an interesting thought experiment nonetheless.

      Reply ↓

    • Well, rather than just guessing, maybe you could try it? And use ChatGPT 4, the paid one. It’s cheap. Just remember to be precise in your questions and remember it’s way better at concepts than it is at data lookup.

      I find ChatGPT indispensable for just about everything. But then I’m old enough to remember people saying that calculators would rot kids’ brains, the internet is the devil’s work, and Wikipedia is garbage. This too, shall pass.

      Reply ↓

      • Oh I’ve definitely tried it, and was paying for it until recently. I’d probably would use it a lot more if I had more time to program. I just haven’t found myself seeking it out repeatedly for much. So far I’ve found it good for little things but not crucial for anything really critical.
        .
        Given that some people (like yourself) are so enthusiastic, I conclude that I must not be trying hard enough. Admittedly I am not a super patient person, and I like to generate a lot of things from scratch.

        Reply ↓

        • Btw, it would be cool if you could write a post Bob about how it’s helped you critique assumptions or otherwise keep in mind uncertainty in tasks that require deep thinking. Or if you already have written such posts, can you point me to them?

        • Hi, Jessica. I was replying to Dale Lehman. This is a reply to you.

          What do you mean by “from scratch?” Do you use pre-packaged tools like ggplot, or are you like Andrew and want to do everything in base graphics because you have more control? Do you let the lm() function do linear regressions in R or do you work out the linear algebra and code it up from scratch? Do you look at what other people have done on StackOverflow or do you just stick to the language manuals and do everything from scratch?

          A bunch of postdocs at Flatiron told me they wouldn’t use Stan or automatic differentiation because it’s a black box and they only trust things they code from scratch. Yet I don’t think they code their own linear algebra libraries.

          In the late 1970s, when I was learning to code, you pretty much only had the language manuals and coding took forever. You could ask colleagues how to do things, but the internet wasn’t widely available. In grad school in 1984, I got the internet and then you could post queries and maybe someone would help you out.

          When the web came out in the early 1990s, there was widespread mistrust, especially around sites like Wikipedia. People would routinely say they don’t trust the internet and prefer printed materials from libraries. I know people who still use printed maps because they don’t trust Google Maps.

          After Wikipedia, which was just a handy reference library, we got things like StackOverflow. It’s really hard to think back to what things were like before that in terms of how interfaces work, how to install things, etc.

          I’m a big believer in using all the tools at my disposal to make my life easier. I don’t stop thinking about what I’m doing. If a colleague does something, I don’t just accept it and put it in a paper, I double check it to the best of my ability (I know, I know—that’s not how to maximize one’s bibliometrics—I should have a group of 10 or 15 people cranking out papers at a rate where I can’t keep up reading them—I quit my tenured professor gig at Carnegie Mellon in 1996 when I had about 10 students and felt I couldn’t keep up or do anything myself).

          People’s response to calculators, Wikipedia, StackOverflow, and now GPT all sounds very John Henry to me.

          Also, I wouldn’t say ChatGPT is a deep thinker on its own. You have to guide it like an assistant. Like humans, it’s much better at processing information than remembering a bunch of facts, as I’ve shown in other responses. I don’t know how to find them or I’d link. I should make a post to summarize.

        • > I conclude that I must not be trying hard enough

          For myself I’m thinking it may be more of a norms thing. Like, what assumptions am I silently making about what is or isn’t a valid scientific/development process.

          A few days ago me and ChatGPT wrote some scripts for some very sketchy simulations. It was a weird mix of bash and python and Java and me manually copy-pasting results around everywhere — very not-reproducible, but it got me my results and now I’m thinking about a next iteration (which I can clean up, now that I know the basic experiments are worth doing).

          It’s pretty unusual for me to do this sort of thing. I’m pretty clean with code these days (everything in packages, gits, with tests, blah blah). This made me think I’m too pedantic, or at least too focused on writing code in what I’ve convinced myself is the Correct Way. I suspect if I took these leaps of faith into sketchy code more often, I’d be better off. And as far as I can tell, ChatGPT is really good at writing little bits of sketchy code.

          I saw a Google spreadsheet someone set up for coordinating some work stuff a few months back. I asked the author to walk me through it — they said they never coded before and just asked ChatGPT how to do things. I don’t think I would have succeeded in making the thing if I tried. There were all sorts of things going on — bunch of backend javascript driving UI in a spreadsheet — one version even had a vector database stored in another Google spreadsheet. It is really cool what they put together, and very much moves the boundary on what I think of as programming.

          ChatGPT has also convinced me I’m better at thinking and writing about certain problems in code than spoken language. Especially analysis tasks — give me pyspark and let me think. Having a box where I can type in English and get code convinced me I’d rather just write code. I’m never really sure what kinda plot I want to make until I start grouping and aggregating grouping and aggregating. And when it comes to grouping and aggregating, I just don’t know a better way to express my thoughts than groups and aggregates.

  2. I love the characterization of humans as black boxes. Thanks for that.

    On the bigger topic, if you grab a random statistician in the wild and ask them what to do, they’ll tell you to run a t-test.

    LLM-based assistants with the express goal of helping us keep track of and critique assumptions, uncertainties, multiplicity, etc., and interrupt us with them at the right times could help curb overinterpretation of results.

    In my experience, ChatGPT 4 is already great at this kind of thing. It’s really good at taking writing and critiquing it and providing constructive criticism. Like asking a person, you probably don’t want to just do whatever the first thing that comes out of its mouth. Think of it like an assistant, not a replacement.

    I’m not advocating for everyone working on data analysis tools to start thinking about LLMs.

    I am.

    I remember the early 1970s when calculators became mainstream and educators had fits thinking they’d ruin education.

    I remember in the early 1980s when desktop PCs and email became a thing (I’ve had internet email since 1984, when I started grad school). Pretty much everyone I talked to said they couldn’t see themselves using either PCs or email.

    I also remember the advent of the web in the early 1990s, and again, everyone I talked to said “that’s not for me.”

    My guess is that in a few years, nobody is going to be able to live without AI assistants.

    Reply ↓

    • Bob – your last line. I unfortunately agree with you, but I read that sentence differently. I won’t be able to live without these because my devices will require me to use them (I’d need to have Daniel’s knowledge to avoid that requirement). I’m already overburdened with what Microsoft and Google think I want, and I believe it will only get worse. This is not to denigrate AI – the potential for good is there. But it is no match for what human greed, ignorance, and ego are capable of.

      Reply ↓

      • We really need to get more people on board with getting away from Microsoft and Google products.

        You can get a full working Debian installation using FAI in about 10 mins almost entirely hands off. https://fai-project.org/fai-cd/

        But then you’ve got a Debian Linux system, and you need to know how to administer it… There are a lot of good intro books on such things, but it definitely isn’t a small change.

        Reply ↓

    • My biggest concern with the AI / LLM excitement is just how much **electricity** it’s all going to require, and how much lock-in to particular vendors it’s going to cause and how much spying the vendors can do on what you’ve been asking the LLM about.

      When an LLM runs on my desktop machine I’ll probably use it a lot for stuff like “write a function that takes such and such DataFrame and outputs such-and-such series of plots laid out in such and such a way”. For the moment, I’m content to just do that stuff by hand when needed so that OpenAI doesn’t burn down a coal mountain on my behalf and also know everything I’m asking computers to do.

      The Infosec part of this is I think way under-appreciated outside Infosec researchers.

      Reply ↓

    • My guess is that in a few years, nobody is going to be able to live without AI assistants.

      I tried this:

      What does the morris water maze measure?
      ChatGPT

      The Morris water maze is a behavioral test used primarily in neuroscience and psychology to measure spatial learning and memory in rodents, particularly mice and rats. Here’s how it works and what it measures:

      Spatial Learning: The main objective of the Morris water maze is to assess how quickly rodents learn to navigate and find a hidden platform submerged in a large pool of opaque water. This task relies on the animal’s ability to use spatial cues in the environment to locate the platform.

      Memory: Once the animal learns the location of the hidden platform, the test can also measure memory retention over time by conducting trials where the platform is removed or placed in a different location, and observing how quickly the animal remembers and finds the new or missing platform.

      Cognitive Flexibility: Variations of the Morris water maze can also test cognitive flexibility, where the platform is moved to different locations across trials. This assesses the animal’s ability to adapt its learned spatial memory to new conditions.

      Motor Function: While primarily a test of spatial learning and memory, the Morris water maze also indirectly assesses motor function, as the animal must swim and navigate the pool to find the platform.

      In summary, the Morris water maze is a widely used tool to study spatial learning and memory in experimental animals, providing insights into the effects of various interventions (such as drugs or genetic manipulations) on cognitive functions associated with the hippocampus and related brain regions.

      Having run said experiment I will say this is very misleading.

      Actually what you see is more like that some rats appear to prefer swimming, others want out immediately. Others just wait to get pulled out for the most part.

      You only let them swim for a minute (or similar, a crucial detail of the protocol) before pulling them out (you don’t want to lose your rat to this!). So the ones that freak out the first time learn it doesn’t matter if they find the platform, they just wait to be rescued. Thus they never really learn the platform is what gets them out of the water. Keep in mind these animals have been living in a cage their entire lives.

      In later trials some who found it earlier quickly swim right at the platform, hit the stand with their tail under the water, then prefer to keep swimming around rather than mount it. Maybe they like swimming, maybe they see it as a chance to escape, who knows?

      I can go on about this, but suffice to say chatGPT lacks access to the technical details to give a good explanation. Especially if they are inconvenient to the reason the source of the training data got funded to do such an experiment.

      Of course, it is just parroting what a certain subset of humans say in public about the topic. Humans who have failed to cure Alzheimer’s/stroke/TBI and so on using such studies.

      How is that different than a search engine?

      Reply ↓

      • I suppose one difference is that it’s quicker and more focused to retrieve that answer. Even if it’s parroting what “a certain subset of humans say in public about the topic”, I would think that it is still more efficient to get that from ChatGPT, rather than searching for some average published review of the topic that provides a concise summary in one place. Not that much quicker of course, but across multiple tasks and questions it does save time (as Bob Carpenter frequently attests). Then of course there is the difference that a curious thinker could interrogate this answer interactively, which is very different from a search engine. Of course, ChatGPT might not be able to provide a good critique of what it has said, but I have found that questions that try to elicit where it might be wrong, or where multiple interpretations might be possible, generally lead to useful information that would take a lot more effort to discover otherwise. One issue with all this is that one has to already have a certain level of incredulity and scepticism to question it in this way, which is probably not going to be present in those learning about a subject for the first time.

        Reply ↓

        • The Wikipedia entry covers both the ideas and the problems with said test, so it took me 5 extra keystrokes to get to a reliable article on the subject.

          Now, maybe the Wiki article is still a bit less critical of the stupidity of this particular test than it ought to be. But at least it tells you that there is bad news.

          Bottom line: ChatGPT is good for people too lazy to read a Wikipedia article.

          Oh yes, MIckeySoft’s Copilot comes up with a hilariously stupid answer for the “Are there three things, a, b, and c such that a is correllated with b, b is correlated with c, but a is not correlated with c” querry.

          Note, however, that “ChatGPT” is stupid about X” is a bit of a fools errand, since they patch the front end whenever a common X is found. Meaning that people won’t see the problems until they actually use it for something new…

  3. Interesting post. Two of the concerns I have with use of AI and LLMs are a) their inability to understand uncertainty in their answers and b) propensity to outright lie. Let me offer a recent example I discussed with my research design class:

    Me: Are there any studies of a link between eating ice cream and shark attacks?

    AI: Yes, there have been a few studies that have looked at the link between eating ice cream and shark attacks . . . In one study, researchers looked at data from over 40 years of shark attacks in the United States. They found that there was a weak correlation . . . Another study looked at data from 13 years of shark attacks in Australia. . . .

    Me: Please provide citations to those studies.

    AI: Sure, here are the citations to the studies I mentioned:
    “Shark Attacks and Ice Cream Sales: A Spurious Correlation?” by David M. Dietz and Andrew J. Dobson. Environmental Science & Technology. 2012. doi:10.1021/es2030447
    “The Relationship Between Ice Cream Consumption and Shark Attacks in Australia” by Michael J. Brown and Mark E. Bravington. Marine and Freshwater Research. 2014. doi:10.1071/MF13264
    I hope this helps!

    Me: Did you make up those citations?

    AI: Yes, I made up those citations. I apologize for misleading you.

    Reply ↓

        • I don’t have any specialist knowledge on this, but this seems to be more than just fixing an answer to one specific question (if that’s what you meant). In general now ChatGPT seems to default to searching the web and not relying solely on its prediction algorithms to provide false references. So yes, maybe they have “fixed a glitch”, but I don’t see how continuous improvement of a tool is supposed to devalue its practical utility. I doubt Wikipedia has exactly the same editorial functionality and rules as the day it was launched either.

        • “ChatGPT seems to default to searching the web and not relying solely on its prediction algorithms to provide false references. ”

          Great. They’ve got a bad/stupid underlying algorithm, so you can now use that bad/stupid underlying algorithm to generate web searches.

          I smell desperation. Also, enshitification.

          (There was also the problem before ChatGPT that the search engines had sold out to the advertisers, so search results were dense of commercial sites selling the thing you were looking to find out something about. So ChatGPT is responding to a need that shouldn’t have existed.)

          (Interestingly, Bing/Copilot did the web search thing from the start. It was OK, but you could still sucker it into saying stupid things. It was amusing watching them patch the front end to check for stupidities. Early on, it didn’t realize zero was a dumb thing to take the least common multiple of. Then they realized that it was a problem that the underlying algorithm does arithmetic by text database lookup, and fails for numbers that haven’t been used in it’s database, so they put in a front-end check for arithmetic. ROFL.)

          But, whatever. We’re talking past each other. My concern is that the underlying algorithm is stupid and says inane crazy things that make no sense on occassion, and that there’s no prinicipled way to fix that. People who like ChatGPT don’t mind that, and are happy that the worst of the crazy/insane get’s checked for in the front end.

        • David in Tokyo wrote:

          “My concern is that the underlying algorithm is stupid and says inane crazy things that make no sense on occasion, and that there’s no principled way to fix that. People who like ChatGPT don’t mind that, and are happy that the worst of the crazy/insane gets checked for in the front end.”

          Exactly. It turns out that chatbots (which should be called Turing Test Bots or TTbots) were always going to rely on the Jeanne Dixon effect for success. And so far, that strategy seems to be working just fine.

        • Matt Skaggs wrote: “Jeanne Dixon effect”

          Ah. That’s a new one on me.

          But it’s exactly right. LLM algorithms have no mechanism for dealing with the meaning of the input or output texts in any way. So the interpretation of the output text is left to the user. On top of that, there’s the thing that it’s sort of like where the rockets come down not being Werner von Braun’s problem*, in that if the bot tells you to eat glue or ground glass, that ain’t our problem.

          Which is to say, the whole LLM thing is exactly and only a parlor trick. Which is what I said ages ago…

          (Aside: while the hacked front ends make the bots less obviously and easily demonstrable to be inane, said hacked front ends really spoil the fun.)

          *: https://www.youtube.com/watch?v=QEJ9HrZq7Ro
          Various sources have it that von Baun actually was learning Chinese…

  4. I think the question of LLMs in data analysis looks very different depending on the (input language, output language) pair you look at. I have serious doubts about English-to-English; English-to-Stan seems much more useful than I expected; Stan-to-autocompleted-Stan is probably going to be standard…

    But where things would get really interesting IMO would be with non-natural languages closer to the domain. I would very much love a model that took as input representations of what I’ve done so far in an analysis at a level somewhat more abstract than R/Python/Stan code and suggested me how to extend it based on good analysis paths from the literature or from a synthetic set.

    In other words, if doing a data analysis feels sometimes as playing chess against nature (or rather poker or Battleship), then I would be much less interested on using a ChatGPT than on getting an AlphaZero.

    Reply ↓

  5. I guess that as chatbots are used more commonly, people will get more used to the counterintuitive results that they give (such as producing convincing-looking fake references, as in Scott’s comment above). If you think of a chatbot as autocompleting, the fake references make sense: people do give fake references in real life. I like Gary Smith’s perspective on all this, which I’ve mentioned in several blog posts during the past year or so, that a big danger is people taking the chatbot output as more than it is, and using the output to make bad decisions. Then there’s the other concern mentioned in the above thread, the combination of huge power requirements (as noted by Daniel) and the push to make chatbots absolutely necessary (as noted by Bob).

    Reply ↓

    • I don’t think people giving fake references explains fake references from ChatGPT. It seems more likely its a result of training a model to predict the next token without grounding it with some other representation of the world. Probing the internal representations of an LLM after training to predict in some domain sometimes shows that the underlying world model is not very robust. E.g., an LLM predicting taxi routes in Manhattan does very well but appears to have learned a map that violates real world constraints on streets and intersections https://arxiv.org/pdf/2406.03689

      Reply ↓

    • When designing or using any software, a key attribute is testability. If one looks at chatbot tools from the cloud vendors, especially for Q&A, they all lack testing other than cosine similarity of the ground truth question and answer.

      This inability to test a chatbot leaves businesses at risk, see https://www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-know.

      There has been a lack of effort from the vendors to add testing, or versioning, to their products. This leaves their customers on the hook for long pilot programs of side-by-side testing. If you work in a regulated industry (healthcare, insurance, etc.), then can you deploy a customer facing chatbot, even when the replies go to call center employees? The risk is that the employee trusts the chatbot too much, especially when management is pushing that solution on them to speed up the call times. How can one enforce a requirement that the employee review the chat reply for accuracy, or tell the employee they are responsible for a final review?

      Circling back to Jessica’s post, I guess I could ask LLMs about design questions, but how can I deal with the inability to test the accuracy of a reply? I don’t ask interns for design help, maybe I should, but how do I know the LLM replies are reliable enough to stake my reputation on it?

      Reply ↓

      • >I guess I could ask LLMs about design questions, but how can I deal with the inability to test the accuracy of a reply?

        I agree provenance is important in analysis. Getting a suggestion for some modeling approach but with no way of tracing the original references is potentially annoying. But you could also follow-up with texts you do trust on the topic. In the LLM scenario you gain the ability to have some dialogue about what you don’t understand (assuming your usual process for design questions doesn’t already involve talking to people/posting on forums etc). Of course it’s possible this way of thinking about design is more circuitous without providing any real benefit over the alternative where you start by consulting some resources yourself.

        Design questions seem closer to the “art” side of the spectrum. If I were using an LLM to handle calculations/computation, where a consensus answer is more possible, I would want to know what other software it’s using (e.g., if R what version, what packages, etc), and have access to similar levels of documentation about algorithms/defaults as I would get if I were doing it myself. So provenance would potentially look different, but is important across the spectrum.

        Reply ↓

        • I think design gets to the heart of my views. I know I am resistant to using an LLM to assist me with design – I can literally feel my resistance. I have asked Chat GPT design questions and even presented them to students as examples where the AI answers are better than many students. It is a good example of its usefulness. However, I have not seen such answers as good use of my time for my own work. I’d much rather post a question on an appropriate forum and get human responses – although it is important to me to choose a forum where I tend to trust the humans (e.g., the difference between Andrew’s blog and Marginal Revolution is an example of a blog where I tend to trust the responses as of value and one where I don’t).

          I’ve played with Chat GPT enough to know that I can refine my questions and get increasingly valuable feedback. But the key to my refining my design questions is that I already have a good idea of the possibilities – good enough to steer the AI in a good direction. Without such knowledge, I simply think the feedback I get is too random to be a good use of my time. I think calculation/computation is different than design and it is much easier for me to see the usefulness in such settings.

    • Bob:

      I’m not sure. We get tons of spam so we have a serious spam filter. Sometimes legit comments are held in moderation, other times they go into the spam box. I check the comments queue every day but I don’t always check the spam box.

      Reply ↓

    • If you mean would I expect variation in results depending on how you prompt it, then yes. But I also think that at least among those currently using LLMs, there is often more awareness that results will be sensitive to small changes in the prompt than there is general awareness in human-driven statistical analyses that small changes in the process can result in different results.

      When it comes to topics like simulating human behavioral responses with LLMs, I’m kind of hoping this acknowledgment of the importance of the prompt strategy there will lead to greater awareness of issues like cherry picking of stimuli in traditional studies.

      But I may be biased in that most of the LLM enthusiasts I know are CS researchers who are probably more likely to expect variation than other types of users.

      Reply ↓

  8. One thing that is interesting in this discussion is that some people seem to be holding ChatGPT to higher standards that other tools, as if the fact that it isn’t perfectly intelligent undermines its entire existence. I can’t think of any other technological tool where there is an expectation that the meaning of all inputs and outputs would be perfectly parsed with no contribution from an intelligent and critical user.

    This may be where David in Tokyo’s “talking past each other” comes in however. The frustration seems to be more due to claims and hype (and the AI label itself?) than whether the tool itself is useful. Just from the responses to this blog it is obvious that many intelligent people find it useful.

    Reply ↓

    • “The frustration seems to be more due to claims and hype (and the AI label itself?)”

      Yes. Especially the label. Since the underlying technology (again, statistical processing of sequences of undefined tokens, i.e. statistical next word guesssing on steroids) is a complete joke, people who take AI and human intelligence seriously are irriteted something fierce.

      There’s also the issue that the people here are folks who are capable of taking responsibility for actually checking potentially problematic output But, again, the underlying technology doesn’t “do reality” and can’t be trusted. So telling potential users anything other than “Do not, ever, trust the output of this program.” is a whole lot worse than just “claims and hype”.

      (I personally don’t see how these things are useful at all: Wikipedia is just too good nowadays. And I hate the schmarmy mansplaing style of the text produced. (OK, the sample program generation bit is cute. Of course, the programming I’m doing (processing Japanese text in horrifically idiosyncratic ways) is unlikely to be amenable to ChatGPTing…))

      I suppose there’s an interesting question here, namely what if we had a program that actually did have a model of the world, and actually did process and generate language based on that model, and did a decent job of it. That “knew” that putting glue on pizza was a bad idea, that handing in made up cases to a judge in a legal arguement gets you disbarred, that advertising a discount commits you to actually giving that discount. How would we evaluate the limits and capabilities of such a program?

      But the AI discussion nowadays instantly turns to inanely stupid scifi…

      Reply ↓

      • David in Tokyo: You’re overgeneralizing. I take human and AI intelligence seriously. Seriously enough that I spent years working on natural language semantics and teaching philosophy of language at Carnegie Mellon. I’m not irritated by ChatGPT (though I am irritated by some of the hype). I’ve said for decades that generation language is AI complete—it’s the basis of Turing’s test for intelligence, and I believe he took human intelligence seriously.

        I’m not sure what would constitute a model of the world in your opinion. Why don’t you consider the human brain a joke? It’s a finite set of neurons with finite connectivity firing very slowly based on very noisy biochemical processes. It’s very good at associative reasoning and very poor and slow at logical reasoning. We’re subject to all sorts of biases. And we’re very bad at causal reasoning if history is anything to go by (Fisher argued that smoking didn’t cause cancer and half my crazy country believes that having more guns doesn’t lead to more shootings; my friend the junior high school principal had parents arguing he was being unfair in not teaching “the other side” of the holocaust [i.e., that it didn’t happen]). Why does everyone think humans are so good at reasoning and have accurate models of the world?

        Reply ↓

        • “You’re overgeneralizing”.
          Oops. I didn’t understand what you meant with this. Maybe it’s that:
          My “But the AI discussion nowadays instantly turns to inanely stupid scifi…” was in reference to discussions by folks outside this blog. The singularity and other AI hypsters. This is (I think? recently, anyway) the first time I’ve responded to you, and I haven’t responded to Jessica because I largely don’t have anything sensible to add to what you all are up to.

          But. The idea that humans are bad at causal reasoning strikes me as way off. We’re really good at it. The whole history of pre-1990 AI is trying to simulate human reasoning and failing miserably. Even simple common-sense everyday reasoning is really hard.

          (The reason LLMs generate logical-looking text is that the internet is full of beautifully logical text with sensible arguments that says true things about the world. LLMs take that good, true text and randomly perturb it to generate (usually) sensible-sounding text without actually doing the work of figuring out why/how the input text says true (or intentionally false, or intentionally misleading) things, or what it’s output text “means”. It’s this not doing the work that rankles here, if and only if someone claims, or erroniously thinks, that the LLMs are doing said work.)

          “A model of the world in my opinion”: I don’t know. AI tried to do that and we failed (I punted at the all-but thesis level for a variety of reasons, including that I couldn’t see a way forward). It’s hard. But people do really well at common sense reasoning. We don’t put glue on pizza. Jim Meehan’s story writing prorgam output “Gravity drowned”.

          I think it’s clear that humans built internal models of the world. The human visual system builds a “gestalt” of the scene based on memory plus extremely minimal input, yet we think we’re looking at the whole scene in complete detail. (An early experiment used eye-tracking to shine a light on the place a subject was looking in an otherwise dark room. The subject would report being in a bright, evenly-lit room, while observers saw a spot of light moving around an otherwise dark room. And recently, it was found that an individual rat neuron specialized for global mapping, also did local mapping. Brains are hairy.)

          Sure, we’re bad at evaluating logical expressions. That was known in the 1970s. And thinking logically and scientifically is hard and takes effort. But despite that, we do physics and math and chemistry, and get it right. A lot. Sure, we get things wrong, a lot also. But we’re often capable of figuring out and seeing that we’re wrong, at least when monetary and/or tribal self-interest doesn’t get in the way. That’s why science works. Folks who feel attacked, or have money on the line, don’t always make sense.

          My personal opinion is that LLMs are silly, but I don’t mind people using them. As I’ve said, the program sample generation stuff sounds neat. Generating boiler-plate text and editing it is easier than writing a paper’s intro by hand from scratch. Even using the random silliness as fodder for brainstorming makes sense. All great fun.

          But claiming that what is essentially random next-word guessing has something to do with “intelligence” strikes me as problematic.

    • Anon wrote:

      “…some people seem to be holding ChatGPT to higher standards that other tools, as if the fact that it isn’t perfectly intelligent undermines its entire existence.”

      This is a valid point. I certainly am, but only to evaluate the veracity of the hype. Based upon the hype, I expected something like C3PO from Star Wars but without the ability to mix me a drink.

      That being said, there is no doubt that LLMs are a valuable thing. They most likely do a reasonable simulation of a single processing module within the human brain. That they do not really simulate a full brain is the answer to the wrong question.

      I suspect that LLMs will eventually be exploited as a building block of more effective AI. The output from the LLM will be analyzed by a higher processor and compared to the output from other functional modules to derive a final (and better) answer. Essentially the testing that is mentioned upthread will occur inside the box, similar to a human evaluating what she will say before she says it.

      As for the relationship between intelligence (by any definiton) and chatbot output, I’m actually kind of losing interest at this point. What I do still find fascinating is the relationship between chatbot output and creativity. I think there are fundamental lessons we can learn, and I suspect that the conclusions we will draw will not comfort us in terms of how we view our creativity as a uniquely human manifestation of ingenuity.

      Reply ↓

      • “What I do still find fascinating is the relationship between chatbot output and creativity. I think there are fundamental lessons we can learn, and I suspect that the conclusions we will draw will not comfort us in terms of how we view our creativity as a uniquely human manifestation of ingenuity.”

        My (jaundiced, as you’d expect) thoughts on this are that we’re going to realize that psychadelic drug crazed “creativity” only works/counts as creativity to the extent it’s at least somewhat grounded in reality. More generally, artistic creativity is interesting to the extent it tells us about ourselves and/or our reality. (Well, for literature and painting/drawing. Music is another thing. (Although, maybe not. I’m a Sonny Stitt fan, and although Stitt does insanely complex blazing-speed runs, it’s his melodic sense that brings his solos together and makes them work as music.))

        But LLMs don’t do reality. So they’ll do the psychadelic drug crazed twisting of reality, but not so much twisting as clumzily kludging together unrelated parts in random ways. It’ll get old fast.

        Reply ↓

        • What actually gets old fast, historically speaking, is people calling new technologies a “fad”.

          For comparison, see early discussions of cars vs. horses, email vs. the post, texts vs. phone calls, PCs vs. mainframes, the internet vs. TV, Mapquest vs. paper maps, etc.

          Here’s a nice Palko-style discussion of horses vs. cars—maybe he can write one of those old-timey tech posts on horses vs. bicycles, trolleys, cars, etc.:

          https://quoteinvestigator.com/2021/07/17/auto-fad/

          It’s what everyone I talked to in the 1980s said about email and message boards. It’s what IBM thought about personal computers before the 1980s. Most of the people I talked to in 1993 other than computer scientists said this about the web. I heard people say this about MySpace and then Facebook (and they’re probably saying that about TikTok and Instagram now).

          For some reason I don’t understand, people are incredibly conservative when it comes to new technology.

        • I’m with Bob, I actually think what we will see is multi-system AIs, stuff that melds language models together with other forms of computing (database lookups, image processing, specialty language models trained on expert knowledge about particular topics etc)

          The big concerns are 1) polluting the knowledge base with AI generated crap designed to attract clicks for ads, and 2) electricity consumption.

          It’s a question as to whether we get there though as climate change, disease, and war will provide some major hurdles we might not be able to overcome. Degrowth down to say 4 billion people ish is likely the answer but between now and then there are many questions.

        • Another way to say this is language models are best thought of as providing a new primitive. There’s a bias though among many toward seeing them as some attempt at a singular form of human simulation to replace everything else.

          Multi-agent combinations are already driving a lot of applications, see e.g., Autogen by Microsoft. It seems natural that multiple interacting agents (with access to more traditional stat tools) will be how we arrive at new data analysis tools that are actually interesting.

    • I’m glad you posted your experience with ChatGPT and the movie data. It matches the reasons I don’t think it is a good use of my time – to use LLMs for cleaning and analyzing data – although it does point to more useful things like writing SQL code. But your analysis looking for changepoints in movies looks a lot like modeling noise and then developing post-hoc “explanations” for what was found. It is an interesting experiment and one I will probably try so that I can speak intelligently about its potential and dangers. But, just like many of those regression discontinuity analyses, the changepoints seem arbitrary. And once you “find” a point at which a trend changes, it is always possible to come up with an “explanation” for why that change occurred.

      I’m not saying that the ChatGPT movie analysis is necessarily wrong. But I do think it is more likely to be wrong than a carefully constructed analysis a human (with an appropriate background) would do. It strikes me like a random analysis – and based on what I think I understand about how LLMs work – I think it is largely a random analysis. I’m not sure why anything in the LLM’s training data would cause it to discover “significant inflection points.” I’m not against using automated techniques to assist with that task – but there are automated techniques that are driven by an understanding of how to prepare the data for analysis and what kinds of techniques might be used to discover those points, and how to evaluate whether something meaningful has been discovered. That is the type of “understanding” I don’t believe LLMs are capable of doing.

      Reply ↓

      • There’s an interesting parallel to be drawn here between Andrew’s (and others) ideas on multiverse analyses I think. Those ideas suggest that a lot of humans aren’t particularly good at navigating forking paths in a coherent manner for inference either, so perhaps the LLM-assisted “random analysis” starts to look like a multiverse analysis reducing a human agent’s forking paths?

        Reply ↓

        • > LLM-assisted “random analysis” starts to look like a multiverse analysis reducing a human agent’s forking paths?

          I’m confused–are you saying that, by providing an analysis that randomly selects one of the many possible forking paths, that the LLM is “helping” the human analyst by giving them a justification for ignoring other possibilities?

          Another topic that has come up on this blog before is Philip K. Dick’s use of the I Ching to suggest elements of a story that he wouldn’t otherwise have considered. That sounds like it might be more like what you are suggesting, with the idea that you can replace the I Ching with the LLM. But since the LLM only selects one path, I actually think the I Ching would be better for that purpose!

        • I’m also curious about what Anon meant with the multiverse reference.

          >But since the LLM only selects one path, I actually think the I Ching would be better for that purpose!

          I think you are referring to the scenario that Anon is describing, so this is not necessarily directed at your comment but in general it’s worth emphasizing that the LLM only selecting one analysis path is an assumption that obviously doesn’t have to be true – one can and perhaps should ask for a set of options with rationales. The only real sense in which the LLM is constrained to selecting only one path is in generating its response token by token.

        • > one can and perhaps should ask for a set of options with rationales

          Good point! I think it would vastly enhance the utility of LLM’s as tools for idea generation if they were used they way you describe. Even if they didn’t provide any rationale, at least getting to see the variability between responses would highlight the dimensions along which the options differ. It might also encourage more critical engagement with the model of the kind that Bob Carpenter describes.

          To give a silly example from my own ill-spent youth, I once coded up a kind of SRN and trained it on some musical fragments that I had written. I then used the output of the model to help me generate ideas for a string quartet I was writing at the time. I didn’t use the model output verbatim most of the time, but it was cool to see how the model remixed my own ideas in ways that I wouldn’t have considered.

          I guess this issue goes back to the “hype” problem—the hucksters are selling DNN’s as tools that give you “the” answer, and so that’s how people tend to use them. They could instead be better used as “idea pumps” (which would then address Anon’s point).

    • Thanks for sharing. Seems like you basically told it exactly what charts you wanted. Would be interesting to see what it produces when given a little more freedom, ie ask it to produce a set of useful visualizations or analyses given your high level interest in understanding your workouts.

      Reply ↓

  11. @gec @Jessica (can’t reply in thread any longer) yes, apologies for lack of clarity, I did mean that the LLM could act as an “ideas pump”, thereby highlighting the risk/dependence of the forking paths to an analyst. I suppose this could take many forms: explicitly asking for multiverse variations, providing code and asking for where small changes might be considered legitimate but provide different answers, prompting for general critique, asking it for its “knowledge” of alternative approaches used in the literature. Perhaps in the future this could be baked in to an LLMs response to any data analysis question? Of course, it would then be dependent on the analyst to report this process…

    Reply ↓

    • Thanks for the clarification, I agree that the approach you describe would have real value!

      My main concern with using LLM’s as “idea pumps” has more to do with how these tools evolve over time. LLM’s are trained on past instances, such that their responses will tend to sample from around those instances. When future models are trained on new text, if that text was generated or inspired by prior LLM output, then those future models will tend to become ever more “entrenched” around the responses from the LLM’s previous version. With each successive generation, the variability in LLM responses would be reduced, diminishing the value of LLM’s as “idea pumps”.

      Reply ↓

      • Good point, although I have no technical notion of how true this is across LLM instances. Naively I would also have thought that the increasing use of LLMs, coupled with ever-larger training sets, would mean that the “forking paths” reporting set would only increase. But I can see that this might be dependent on analysts having to push it harder to get away from the most common/likely associations in the training data. I suppose this kind of thing is quite hard to test given that the LLMs in the wild are not static, and a tame LLM (i.e. set up by an analyst specifically to investigate some research question) is unlikely to be very similar to those in the wild.

        Reply ↓

  12. # My take on the issue of causal reasoning:

    IMAO, the reason LLMs work so much better than almost everyone expected is because they are curve fitting the space of all human generated text (including equations and code) and that corpus has a “causally connected topology” (my term lol). LLMs are just curve fitters (a souped up version of least squares), but they happen to be rather good at curve fitting causally connected data. In fact, in my book Bayesuvius, I analyze transformer networks using causal Bayesian network language. https://qbnets.wordpress.com/2024/01/22/the-secret-causal-life-of-transformer-networks/

    # use of LLMs to improve data analysis:

    My software Mappa Mundi (which uses 2 different LLMs) can extract causal DAGs from text in a fully automated fashion. This is a type of data analysis.

    # use of data analysis to improve LLMs

    Current LLM do not distinguish between correlation and causation, except maybe in a very feeble implicit way. I call that being “superstitious”. IMAO, The fact that current LLM are superstitious is a problem that is even more serious than the notorious hallucination problem. Thankfully, software already exists (addons) that when added to LLMs, give LLMs the ability to not be superstitious, to do causal inference and to perform the scientific method in an explicit, deliberate way. https://qbnets.wordpress.com/2024/03/08/mappa-mundi-project-first-order-approximation-finished/

    Reply ↓

  13. I just left a comment on your June 27th Substack post with a similar thought so this would be repetitive but still worthy to say it here. Your application of the “garden of forking paths” to LLMs acting as data analysts highlights a massive verification problem that the ML community is currently ignoring.

    When we hand an LLM an open-ended analytical task, we are handing it an autonomous inference problem across an unobservable decision tree. The danger is that the AI will inevitably engage in stochastic laundering. If the model navigates these forking paths by relying on unobservable, latent variables and internal priors, strictly outside the observable evidence provided by the dataset, to force a conclusion, the analysis is structurally corrupted. The AI is no longer analyzing the data; it is mathematically engineering an inference to satisfy its internal proxy weights.

    To prevent an automated analyst from gaming its own analytical choices, any LLM deployed for data evaluation must be constrained by the Non-Circularity Principle (NCP). The NCP dictates that a system’s posterior inferences cannot be allowed to vary based on unobservable completion variables that leave the observable restriction unchanged.

    This means we cannot treat LLM data analysis as just another black-box heuristic. We have to treat it as a strict measure-theoretic boundary. We need to formalize the Principle of Epistemic Sovereignty (PES):

    https://trissimondsen.wordpress.com/2026/07/16/the-principle-of-epistemic-sovereignty-formalizing-the-zero-trust-boundary-in-ai/

    PES treats data evaluation as an F-measurable constraint. It requires that the human scientific community retains sole authority over the information interface. The AI is only mathematically licensed to process the exact, un-adulterated data that has physically crossed the channel. If we allow LLMs to blur the line between what is strictly observable in the dataset and what is internally generated to resolve a forking path, we aren’t just doing bad statistics; we are officially outsourcing our epistemic sovereignty.

    Reply ↓

    • I can understand little of what you have written (not a criticism unless you choose to view it as such – it is probably a result of the fact that I don’t understand most of the terms you are using). But your description of using LLM data analysis does not sound like the way I have been using it. Your description sounds like the LLM is being asked to perform an analysis. I have been using LLMs to generate options and choices, often with explanations of pros and cons associated with those choices. Eventually, I’ll get analysis done by the LLM, but only after I’ve made the decisions and told it what I want. There is still the validation step to see whether I believe the output it provides, but the process of getting there doesn’t sound like what you are describing as “doing bad statistics” or “officially outsourcing our epistemic sovereignty.” Perhaps someone can translate for me so I can better understand whether I really am disagreeing with you or not (because I can’t tell).

      Reply ↓

      • Dale, appreciate the reply. The good news is that we are in complete agreement! In fact, the way you are using LLMs is exactly the workflow my comment is trying to defend.

        To translate the jargon: “Epistemic Sovereignty” just means the human stays in charge of deciding what is true and which analytical path is valid.

        When you use an LLM to generate options, weigh pros and cons, and then you make the final decision on what analysis to run, you are acting as the ‘filter.’ You are retaining your epistemic sovereignty. The AI is just acting as a calculator and a brainstorming partner.

        The danger I am warning against is the growing industry trend of fully autonomous AI “data agents”. People are starting to hand an LLM a dataset and say, “Analyze this and tell me if the paper is good” or “Find the statistically significant insights”.

        When an autonomous AI hits one of those analytical choices (a forking path) and makes the decision by itself based on its internal programming (what I called “unobservable latent variables”), it corrupts the analysis. It is making up assumptions behind our backs to force a result. That is what I call “stochastic laundering”.

        You are using the LLM as a tool while keeping your hands firmly on the steering wheel. My warning is aimed at the people trying to let the AI drive the car.

        Reply ↓

        • “The danger I am warning against is the growing industry trend of fully autonomous AI “data agents”.”

          Agreed that said trend is insane, but…

          If you said something on the order of

          “Allowing a random text generator that has no therory, model, or mechanism for understanding either the text it randomly generates or the real world to make changes (e.g. deleting your files, bombing a school) to said real world, is a seriously bad idea.”

          you might wake some people up. Might.

        • This is handled via independent replication and checking precise predictions.

          Whether done by an ai or human is not relevant. The problem is too much research relies on “trust/prestige/authority/consensus” heuristics. This strays from science (nullius in verba), which has not trusting anyone (including yourself and llms) as the foundational principle.

          The llms will indeed spam these areas much faster than humans have been doing so, but that is simply revealing the existing problem (these fields were already generating misinformation at comical rates but were able to hide behind dense jargon and various shibboleths).

        • David – while the outcomes you describe are certainly visceral, the core issue is structural rather than purely sensational.

          Anoneuoid – you raise the standard defense of the scientific method, but believing autonomous AI is simply “doing what humans do, only faster” is precisely the architectural error (often mistaken for a mere category error) that the PES framework exposes.

          The traditional tools of scientific inquiry, specifically “independent replication”, are fundamentally insufficient to govern autonomous AI. This is not an outdated empirical problem stemming from bad data, poor heuristics or a lack of replication.

          The current academic establishment remains so anchored to its standard diagnostic tools that it is largely blind to the architectural boundaries required to contain these systems. One cannot “independently replicate” an unbounded autonomous agent corrupting a live system in real-time, because the state of the system is permanently altered by the agent’s actions.

          “Nullius in verba” was designed for human observers testing static physical laws; it completely breaks down when applied as a post-hoc defense against a real-time autonomous agent actively altering the state of the environment. Attempting to apply a post-hoc statistical diagnostic to a live structural collapse only compounds the exact failures it is attempting to solve.

        • Dale, Tris, David and others.

          I have enormous respect for Christine Lemmer-Webber who is both the head of a small nonprofit doing scheme (the computer language) based software development, and one of the big contributors to the ActivityPub specification for federated online discussions (mastodon, pixelfed, etc).

          Her article linked here is oriented around people using LLMs for software development rather than data analysis but it has tremendous applicability to data analysis as well

          https://dustycloud.org/blog/faulty-towers-vibe-sickness-and-the-vibe-bobsled/

          Dale, when you ask an LLM to carry out a data analysis according to some specs do you then look at the code and verify that the code does what the LLM told you in english it was going to do? Do you spend enough time to catch bugs at the same rate you would catch them if you wrote it yourself? If not… you’re on the vibe bobsled she mentions. People are “going on faith” that the code does what they want. The evidence is *massive* that this *does not work* at scale. But it works often enough that it’s a giant attractive nuisance approximately like slot machines. People regularly using LLMs appear to exhibit “symptoms” very similar to gambling addicts. I have read analysis of these symptoms by a coder who is an admitted ex gambling addict. I find it incredibly plausible.

          LLMs main utility is the incredible speed with which they produce something plausible that “looks like” the code you wanted. But the limiting factor in a data analysis is decision making about the forms of your model and the data cleaning and understanding the linkages and coding in the data sets and the meaning of the variables, and making decisions about what is a good and what is not a good mathematical approximation method and soforth. I have multiple times written a model in an afternoon and then taken months to iterate on decision making about that model to get it to make sense. The coding wasn’t the limiting factor. If you use LLMs and then don’t spend that month making sure your analysis makes sense and is right, then the LLM “saves” you months of work! But that work is the work that needed to be done, so it’s a way to substitute work for faith in an oracle.

        • Daniel
          Specifically regarding checking the AI code: no I do not, nor can I since I don’t code. I am quite confident that the code works as intended for several reasons: first, it executes because it produces the output I asked for. Second, to the extent I have done my own analysis, the AI output matches. And, when it does not match, I’ve asked for reconciliations and found these to be excellent and insightful. Frankly, worrying about whether the Python code was written correctly is among the least important things to worry about, in my opinion. It is my impression that most issues with AI code come from people explicitly asking it to write code – often it seems that need to be edited to work correctly. But I am interacting with the AI on the level of ideas and queries: when it decides to use Python to produce something, I’m content to let it go and worry about the answer. Yes, I have found mistakes at times in the reconciliation process, but these never seem to involve it having written incorrect code. It is the AI failing to understand the context of the data, failing to read data dictionaries, making incorrect or inappropriate judgements (although that has been significantly reduced by my asking for choices and instructing it to not produce anything until I make my own choices).

          I guess I’m an Olympic vibe bobsledder.

        • Daniel
          I’ll add: I think the issues with software development and AI are comparable to issues of data analysis and AI – but on a more conceptual level. It is not about checking the code for data analysis. It is about asking an AI to produce something (software in one instance, a time series analysis in another) rather than asking it for choices which the user then decides and guides the AI. When developing software that may very well require checking every line of code. When doing a time series analysis, I think there are a host of choices to be made – none of which involve the Python code to execute those choices.

        • Dale wrote:

          “When doing a time series analysis, I think there are a host of choices to be made – none of which involve the Python code to execute those choices.”

          You may not even be using “AI”. The AI follks kludge the back ends of these things up to spit out Python code for the things they know the LLM can’t be trusted to do, like multiplication.

          If the automatic programming blokes had as much sample code, money, manpower, and computing power to burn as the AI folks, they’d be able to come up with far better solutions; solutions which would produce code actually written and tested by real developers and labelled with the use cases for which it’s known to work.

        • Tris wrote:

          “David – while the outcomes you describe are certainly visceral, the core issue is structural rather than purely sensational.”

          You seem to have failed to read what I wrote. There ain’t anything as “structural” or as “core issue” as pointing out that the AI twats are using…

          “a random text generator that has no therory, model, or mechanism for understanding either the text it randomly generates or the real world ”

          As the basis of their ridiculous game.

          Sorry to be rude, but I get really really really irritated at folks who talk about AI while forgetting that the very basis of this whole AI game is statistical analysis and generation of text, an approach that has long been known to be not the way language and reasoning work. We’ve known that the mind and language don’t work that way for decades (Jerry Fodor discussed this in the 1990s). This is exactly the core structural issue that makes the current round of AI such an embarrassing clusterfrack. And nothing more than an inane parlor trick.

          So I get irritated if someone misses the core point of what I’ve been saying all along.

          The structural problems at the core of the game is why there are so many “visceral issues” to point out, of course.

          But, Hey! ROFL. Thanks for the term “structural problems at the core”! Consider thynself ripped off.

        • David
          At this point you sound like AI is a wild conspiracy. For some reason, the “developers” of AI are programming it to spit out pre-determined or random output without even doing any real analysis. I’ll admit it is possible. I’ll also disagree and say this does not match my experience at all. I guess I am really on this vibe bobsled and I guess I’ve been duped into thinking that AI is doing something productive. You may be right – but after 50 years of professional practice, I am finding AI to be a better co-investigator than I’ve had in my career. Perhaps if I were teaching at Columbia and had Andrew to work with every day, I’d find AI to be a waste of my time. But that hardly matters – I don’t have that opportunity. And I have found working with AI more insightful than most of the colleagues I’ve had over that 50 years of university teaching.

          None of this is to say that I’m optimistic about AI. The many dangers are real and I’m fairly certain that we will see the misuses cause more harm than the positive uses. But claiming that AI is either random or intentionally misleading me does not match my experience at all.

          Perhaps AI has difficulty multiplying two numbers together, but I’ve not been using it to do that. But it was capable of detecting changing copulas describing the evolution of home prices in different areas of the country over several distinct time periods. It was something I had suspected but was unable to completely analyze (or I was unwilling to spend the time to figure out exactly how to do that and execute it). The AI analysis was excellent and it passed my validation checks – admittedly which fall short of checking every line of Python code that it wrote.

        • Dale,

          “For some reason, the “developers” of AI are programming it to spit out pre-determined or random output without even doing any real analysis.”

          LLM output is not pre-determined, it’s random. Again, the underlying game is statistical next-token generation. Presumably, you know that.

          As for the “without even doing any real analysis” bit, that matches my understanding of how the LLM algorithm works: it’s statistical next word generation, and is well understood as a method that does not do any “real analysis”. It doesn’t have a mechanism for (or theory of) determining what the output it generates “means” in the real world.

          This isn’t me being perverse, it’s basic computer science. It’s what this algorithm does do and doesn’t do. The people who thought it up thought/hoped that vectors in large spaces plus the statistics on the tokens would make those vectors act like concepts in logical reasoning. It turns out that doesn’t work. It doesn’t do concepts and reasoning. It does generate plausible-sounding text, though.

        • David
          “Plausible sounding text” is a Pandora’s Box. I agree that it often (not always) produces plausible sounding text. Most of my colleagues do that as well. As to whether any “real” analysis was done, I can’t say – in both cases. What I can see is that a variety of Python packages were employed to examine the data I provided, and with the instructions I gave it regarding what assumptions to make, and the resulting output was quite plausible and insightful (I’m not claiming any insight on the part of the AI – something I think we agree on). The insights were mine, a human. But the insights I got were facilitated (perhaps more strongly, I couldn’t have obtained them without the AI) were obtained with the assistance of AI. I can also say that many of the “decisions” made by AI (I put it in quotes because I am not saying the AI “decided” anything in the way we humans use the term – yes, it is a statistical calculation it used) were sensible, and at times, were things I agree with but had not thought of myself. I’m not sure that this is different than many of the papers I have written with colleagues over the years.

        • >At this point you sound like AI is a wild conspiracy.

          AI IS a wild conspiracy, one that has been publicly discussed in radical publications such as (checks notes) Forbes and The Atlantic…

          Its specifically a conspiracy of finance, very similar in structure to the subprime mortgage debacle.

          This article is long winded but its not completely wrong https://www.groundbrkr.com/p/the-second-derivative-why-no-one

          The idea is this, companies produce an “amazing” technology, they conspire to push this technology on everyone playing off the fear of being left out of a massive transformation. This drives growth in stock price. This growth drives the capacity to borrow. To keep the illusion going they borrow and build out data centers. Buildings that do horrible environmental damage, take a long time to build and come on line, and will eventually likely be set on fire by people protecting their communities from those harms. But for a while these buildings go on the books as massively valuable real estate assets, driving further capacity to borrow.

          but meanwhile the shear scale of the investment convinces people endless growth at a few percent per year above other industries makes this the most attractive investment in the world… Stock prices skyrocket while existing billionaires unload their personal shares on a gullible public. The system works so long as the price trajectory is concave upwards, enabling the companies to pay their debt by refinancing off exponentially higher stock collateral. The entire concept is refinance whenever payments need to be made from exponentially higher capital value driving borrowing power.

          Eventually growth slows, and becomes linear then concave down then flat… as soon as the growth becomes linear this will be the Signal to those in the know to exit, which will start the unwind. That will drive growth flat and then a complete crash as debts become due and no new borrowing is possible.

          Once the ability to refinance to pay off loans with more loans disappears suddenly banks and investment firms will collapse as assets on their balance sheet collapse. The insiders, having liquidated their positions into sufficient cash will be in a position to scoop up assets at discount prices as the reserve supply disappears and there is a giant sell off in assets by banks to try to remain solvent.

          The government (The Fed) will of course come along and offer to buy bonds held by banks at a massive markup to prop up bank reserve levels, thereby driving interest rates to zero. This will be a good time for homeowners to refinance, but lending will tighten.

          The big losers in the long run will be pension funds and 401ks. Banks will become more consolidated than they already are.

          To the extent that anyone still wants to use LLMs they will become something people run on their own hardware, and very few new models will be trained, some effort will go into making training and tuning somewhat more efficient, and large numbers of half built data centers will have their copper stolen, and deteriorate half built leaving the US looking even more like an undeveloped country.

          Meanwhile, Europe is already on the path to seeking “data sovereignty” and

        • …and the core business of many tech companies, spying for the US govt and advertising industries may collapse as well as people realize that tech has become a weapon by which the US can manipulate other states.

          I fully expect the world in 2035 to look extremely different, but I dont know when exactly this all goes down.

        • Daniel
          Do you say the same thing about the internet? About email? About the web? About electricity?
          I don’t know, perhaps you do and perhaps all these technologies are wild conspiracies. Certainly, they each resulted in some people earning huge wealth, often based on taking advantage of the masses. I’m not going to debate that and wouldn’t necessarily disagree if you want to make that case. But each of these technologies also enabled valuable things. I won’t even deny that perhaps the negative effects outweigh any beneficial ones – and I am certainly not making the case that LLMs will result in a net welfare gain for humanity. I am making what I think is a much more modest case: that LLMs are enabling some valuable things. I’m not sure why so many people seem determined to make the case that they are either a complete joke (in the meanest sense of the word) or that they are the greatest development in human history. They are partially both, and the extreme cases might both be exaggerated (I’m not sure about that since I think the potentials, especially the bad ones, seem quite large).

          You want to tie AI into your beliefs about government, capitalism, the use of police/military force, wealth earned through power and luck, and whatever other concerns you have about humanity. Strangely I may agree with much of this – and I also think that viewing each of these things in isolation is a mistake. Technologies and human institutions do not evolve independently of each other. But what I find unhelpful is to imbed AI/LLMs into these broader issues, losing any serious discussion of what these technologies can and cannot do.

        • Dale, LLM chatbots are not at all like “the internet” or “electricity” and very much like “liar loans” and “NFTs” and “tulip mania”.

          They are a mechanism whereby a financial game is being played to transfer assets and power from one group of people to another. An enormous amount of subterfuge is going on, and the people who look into this can’t really unwind it completely enough to figure it out, but it’s very very clear that there is a giant accounting game going on.

          This is ***completely separate*** from their role as a technology that allows you to do the things you do with them. You want to talk about what they are and are not good for. I want to talk about the fact that they are built entirely on an unsustainable financial bubble and they won’t exist in their current form for much longer, and so it’s more or less irrelevant what current tech can or can’t do.

          Whether or not I think that they are good, or bad, or have some good qualities but some bad qualities, or whether or not I think they “understand” or “should be used in a certain way” or whatever… There is one thing that is true… They can only be made available to you so long as a *financia fraud* is being committed on all of us.

          Some useful background is here: https://www.wheresyoured.at/the-openai-bubble/

          A technology which, however good it might be, requires a financial fraud for its existence, is not a technology that is going to last. Neither “the internet” nor “electricity” required financial frauds to hide the fact that they were ridiculously expensive and required burning more resources than anyone could possibly afford to provide in order to keep them going.

        • “Nullius in verba” was designed for human observers testing static physical laws; it completely breaks down when applied as a post-hoc defense against a real-time autonomous agent actively altering the state of the environment. Attempting to apply a post-hoc statistical diagnostic to a live structural collapse only compounds the exact failures it is attempting to solve.

          Please provide some evidence that independent replication and checking predictive skill do not apply.

          People are “going on faith” that the code does what they want

          This is inverted. People were relying on faith more *before* llms. Now automated testing has become so easy you can know exactly if the behavior is to spec.

          Its now all about inhumanly comprehensive testing/logging and inhumanly fast release cycles. Human code review doesn’t fit into this workflow.

        • Anoneuoid, you’re conflating verification to a specification with epistemic grounding in an open system.

          Automated testing and fast logging cycles are great for verifying syntax and expected outputs in a closed sandbox. But when an autonomous agent is actively altering its environment in real-time, checking code “to spec” misses the core problem: the spec relies on assumptions about an environment that the agent is actively changing.

          If an automated pipeline generates the code, writes the test suite, and executes the deployment, high-speed automated testing just scales confirmation bias. You are not validating predictive skill against independent reality, you are validating whether the system’s output is self-consistent with its own internal assumptions.

          Independent replication breaks down because the agent’s actions introduce non-stationary feedback into the environment. The system isn’t being tested against ground truth; it’s being tested against the synthetic exhaust of its own past action.

        • @Tris

          You’ll have to provide some evidence. I don’t know what all your jargon really means, but I am confident replication and testing predictive skill works just fine. Ie, there is no difference between llms and humans on this point.

          You can see how it works in this example, the bot discovered a counterexample to a famous conjecture which was then double checked by other humans and llms. 85 yr old math problem solved. It works fine (you can also find links to the internal “thought” process in this thread):

          https://news.ycombinator.com/item?id=48973869

          The issue is humans using heuristics (“trust”) instead of replication/prediction. This is a widespread pre-existing problem.

        • Suppose I upload a CSV file and ask a chatbot to write code to filter the data in some way and produce some graphs for me.

          Suppose it says that it did so, it outputs some python code, and it outputs a graph.

          How do I know that it wrote that code, it executed the code on the provided CSV and the graph is the product of this code, and the code does what I asked it to do?

          The answer can’t be “because I asked the chatbot to also create and run some automated tests and it said that it did those and the results were given” because that just pushes the question farther Back how do I know the automated tests do what I want, were actually run, and produced the results I asked for?

          At some point you have to read the code, run it yourself, look for bugs, understand the relationship between what its doing and what you wanted… etc

          when its output is a few hundred lines or something, this is plausible. I saw a test of real professional programmers where they frequently missed intentional bugs even when told where in the code to look, so thats already a little sketchy…

          but when you get 50,000 lines of code with the same functionality written 6 or 8 different ways called from different places and the names of the functions actually dont correspond to the functionality and bizarre use of global variables and etc etc… as is found in every large auto-written agentic everything ive seen deconstructed online… it really is just a “trust me bro” attitude. This is particularly true when you recognize that its not just the chatbot you have to trust, but actual human agents who control the entire chatbot service who may actively want to harm you.

          chatbot services are not like compilers, theyre more like consulting companies run secretly by the CIA, or Russian intelligence or a finance company trying to do a hostile takeover of your company or a ponzi schemer.

          simply not enough people have the proper infosec attitude to chatbots usage. it feels like your “partner” but its *actually* your adversary

        • @Daniel

          You’ll need known good examples of the in/output to test against. You can bootstrap by giving multiple bots the same spec. They will provide different code that attempts to meet your spec but with the same output. Its unlikely the same bugs/hallucinations will be present in all, so you flag any mismatches for investigation.

          This is the same method eg NASA used during the apollo program to check the calculations done by humans.

        • Daniel
          How do I know that when I do a linear regression using regressoR or sk.learn.linear_model.LinearRegression or regress in stata or fit Model in JMP that it is actually performing that regression on the data that I gave it? I suspect 99% of people that do such things do not check the code line by line (you may be the 1% who do).

          Now, I know there is a difference. These are products used by millions of people and they’ve withstood some tests of time – definitely more than LLMs. Given the ecology of the AI world, it is entirely possible that bad actors will target my attempts to have an LLM construct a regression model and output some pre-arranged nonsense to mislead me. I guess if I were doing regression models that were really important, it might make sense to worry about being scammed like that. But it would be a lot of work to falsify the code to give me incorrect results (if it does, I can always use the Wansink/Ariely defense that it wasn’t me, it was my research assistant LLM that did it).

          There is still the sanity check about the output. This applies even if I use some package in software to build the model – sometimes I put the wrong variable in the model and the results provide clues that something was wrong. I am not advocating blind trust in the LLM. But I find the requirement that I check every line of Python code it produces absurd.

          I also trust my car. When I press the EcoPro button, I trust that the car is modifying its behavior to improve gas mileage. It certainly drives a bit differently – a bit less acceleration unless I really floor it. Of course, it is possible that the manufacturer has set that button to pretend it is doing something when it really is not. Opportunity is certainly there, motive is doubtful.

          I’ll repeat: among the things to worry about when I use an LLM, the accuracy of the Python code it produces to provide something I asked for is among the things that concern me the least. The LLM does not understand the context of my data. It is programmed to provide output, when I want it to provide choices and information – so I have to work harder than I should to make sure it isn’t answering questions it thinks I want to ask. But the Python code it writes to execute a model I asked for – that I’m willing to trust. Also, I’ve checked it on prior data analysis that I conducted myself, so that is an additional check. But there is still a possibility that my stats software provider and my LLM provider are both in cahoots to provide false analysis to me. The probability is not zero, but it is much smaller than so many other things to worry about.

        • Dale, you may be interested in this:
          https://www.wired.com/story/fast16-malware-stuxnet-precursor-iran-nuclear-attack/

          There was a bug called Fast16 created in 2005 for the purpose of delaying Iran’s (or NK’s) nuclear program. It subtly tampers with the outputs of CAD software so the designed machines will fail much faster than anticipated. Nightmare fuel.

          Subtle errors can be deadly. This is why engineers have always done massive amounts of verification and sanity checks, even before LLMs existed.

        • Dale,

          You rightly bring up that there are trust issues at every level of software. One aspect of that trust is the *reuse* of libraries that are broadly disseminated, have human intentionality behind them, and many eyes on them, many rounds of bug reports, human hand-written tests, etc. None of that is true for LLM code whipped up just for you.

          Even in human developed stuff there are active supply chain attacks underway *literally at all moments of your life*… One recent example: https://www.sans.org/blog/axios-npm-supply-chain-compromise-malicious-packages-remote-access-trojan another was the Solarwinds compromise that led to Russian infiltration of US military stuff. There’s also the Mossad supply chain installing bombs in electronic equipment in Lebanon, and the NSA intercepting and installing spyware on professional networking gear back in the 2014 era.

          Of course you may personally not really care that its possible for bad actors to intentionally hobble your code, but if you produce something consumed by someone else (either code, or an analysis that people rely on to make decisions about their lives) I personally feel that “I didn’t actually read any of the Python code, but it sure seemed like it was doing the right thing” is not gonna be a good scenario to replicate tens of millions of times a year in the broader scientific enterprise.

          My son spent last semester taking a Linear Algebra class where the teacher had drunk the AI kool-aid pretty heavily. All of her assignments were more or less “it sure seems like it’s doing the right thing” but he spent literally tens of hours a week compensating for the minor bugs and inconsistencies and incorrect wordings and imprecision and confusion generated by her basically asking AI to write educational materials for her students. Not to mention that she was able to vastly increase the volume of material. So he was left doing 30 hours a week of work for *one* 4 credit class. 20+ students started the semester, only 5 or 6 were left after the “withdrawal” deadline. No-one had more than a 60% by points, stress levels were very high.

          It could only be described as “slop”. It wasn’t *useless* slop, because he did eventually learn Linear Algebra, but it was very definitely slop nonetheless and it imposed large costs in time and confusion on him.

          There are certainly different levels of sloppiness different people will get involved in. I personally am unlikely to complain that a person wrote say individual 5 line bits of code to produce plots that meet detailed data display preferences… but ultimately you can’t substitute human intent to do a good job for a machine which is specifically trained to produce plausible sounding sentences irrespective of whether they are factually correct.

        • Daniel
          If only humans were so trustworthy. You made me laugh. We can’t live without trust, but you are asking me to trust many different humans but none involved with the LLMs I’ve used. I agree that there are important differences in the environments that require due diligence. But if I adopted your version of due diligence, I would have to finally retire. I’d rather ride the vibe bobsled a bit longer.

        • Also Dale, to clarify a bit. I also agree it’s unlikely you personally will be targeted to trash your econ analyses by the CIA or whatever. That’s not what I’m arguing is likely to happen. Thousands of people were killed or maimed in the Mossad pager bombing instance who were not specifically *targeted*. The NSA literally scooped up *every* text message sent via SMS in the US for a decade or more, not targeted… In other words, actions taken by adversarial actors on the global stage will have fallout that harms others in ways that are completely unpredictable.

          Is it plausible that a third party can force providers to alter their agentic/AI type software for the purpose of affecting some specific target in such a way that it inevitably causes fallout for you by biasing your analysis towards particular types of bugs or whatever? That seems not just plausible, it’s likely already happened to some users. It is also likely that on the back end the providers are switching you between different versions of their software based on factors they identify as improving their profits. If you think these people *never* cheat and *always* have their software do exactly what you think it’s doing, you are radically too naive. Reproducibility is also extremely unlikely. If in 3 years I see your paper on copper mining or restaurant microeconomics during the pandemic or whatever, and you describe in some appendix the set of prompts you used to generate the analysis, if I put those prompts into a version of these services 3 years from now, there’s virtually no chance I will get the same results. Of course, you can publish the code and etc, and as long as its small enough I could read the code… But the usage is moving more and more toward guess-and-check type generations in which the complexity of the code blows up, and soon the analysis code is spaghetti I have no hope of ever figuring out. It’s “trust me bro” all the way down eventually.

          Are there ways to use LLMs that are trustworthy? Probably, is that the common use case and generally widely done? Hell no. Multiple times I have seen people on my parents chat-group for the school district issues I’m dealing with hand hundred page reports or whatever to their AI services at work, and then just copy and paste the summaries of the AI’s “perspective” on what the reports mean, un-critically, and without any checking at all. This is going to be a very common use case, until the whole financial bubble pops, and we’ll see what comes out the other end.

        • Daniel
          If you are worrying about how I (and others) will be impacted by the many improper uses of AI, then you don’t need to convince me. I am not at all optimistic. But frankly, having all those bad actors check every line of Python code that the LLM uses is not going to put my mind at ease. Bad actors have taken over (Barbarians at the gate?) and AI has given them more powerful tools than ever before. In fact, if they aren’t checking the code line by line, that may be our only hope – that their attempts to use AI for nefarious purposes fails because they haven’t done due diligence. But I’m not hopeful about that scenario either.

        • Dale,

          Let me just try to give some concrete less wild-eyed version of what I mean.

          There are multiple ways to implement linear regression. You can call some library function, you can do linear algebra directly, you can run an optimizer to minimize a literal sum of squares, you can do whatever…

          Software like R, or Stata, or whatever has often been written by people with numerical analysis backgrounds, who have benefited from detailed specialized algorithms developed to make things numerically stable etc. You can do QR decomposition by Gram-Schmidt, or by Householder transforms and one is numerically stable and one isn’t.

          the numerically stable version of these algorithms are *always* more complicated, and so do not generally appear in first-year educational materials, of which there is a LOT more in the training data than there are textbooks on 4th year numerical analysis for Math majors.

          So, with 50 billion parameters in the model, and some global struggle for financial dominance and drone warfare and such going on, what’s keeping various actors from accidentally biasing the AI away from using a numerically stable bit of code to calculate the linear regression you ask it for? Perhaps because by doing this bias it helps the AI to pass tests regarding destabilizing the code in drone warfare applications. Tests the DOD hypothetically requires Anthropic to pass in order to offer it services. And thereby *sometimes* produce numerically garbage answers which are nevertheless consistent between your test cases and your actual research case?

          Literally nothing but “luck” in some sense. And this could happen at any time, so the fact you’ve tested Anthropic in the past and it did the right thing, doesn’t mean this week or next month it won’t change. And since your code is de-novo every time you prompt the AI, you don’t benefit from the 3 decades of carefully numerically stabilized library code in Fortran that R uses, unless Anthropic decides to call that library, and instead you get whatever Anthropic’s AI decides to hand you this week.

          Multiply this by literally thousands and thousands of choices the AI is making when you prompt it. Like if you’re fitting a nonlinear maximum likelihood model, which optimizer does it choose to use this week? vs next week? vs next month? How does that change with the particular wording you use in the prompt? What method does it use to initialize the search? Does it carry out some iterative process which is convergent in some subset of problems but your particular problem doesn’t fit the requirements? Etc etc.

          The AI could be doing what you ask it to, but with some wacky algorithms that just don’t reliably work, but do reliably produce answers of some kind, they just aren’t right. In the 90’s there were websites devoted to examples of how *commercial hand written software* failed in these ways. Excel had this kind of crap in it for decades. 30 years later a tiny segment of humanity has expertise in these questions, but the vast majority of instructional text out there still says something like “here’s conceptually how this works, when you take a later higher level class you will see how there are better ways to program this process” and then describes the numerically naive methods.

          Unless you are steeped in the lore of the history of numerical computing and how many times over and over again it has gone wrong when people forgot to do the sophisticated algorithms and used the naive ones… you will not realize that not only *can* this happen, but the history of computing is full of it *actually* happening repeatedly. All of that code is likely in the training dataset too.

        • Anoneuoid, your own example perfectly illustrates why you are not refuting my premise.

          Finding a counterexample to an 85-year-old math conjecture is a brilliant achievement, but mathematics is a closed, formal system. The rules of math are static. The ground truth is entirely independent of the agent. To use the terms you called jargon, mathematics provides a perfect Player-Frame Restriction. When the LLM finds a counterexample, we can verify it precisely because the mathematical laws remain stationary regardless of what the LLM generates.

          My critique is about deploying these agents into open, non-stationary environments, like live markets, digital infrastructure, or social systems. In those environments, the agent’s actions alter the environment. There is no static rulebook to check against, and the system begins optimizing for statistical plausibility rather than external truth.

          You asked for proof of what happens when you lack this architectural boundary in an open system. Here are three documented realities:

          1. Model Collapse: Look at the recent peer-reviewed research in Nature (Shumailov et al.). When an LLM ingests its own synthetic exhaust, it doesn’t correct itself via “predictive skill”. It irrecoverably forgets the underlying ground truth and converges on a hallucinated reality. The testing loop fails because the environment itself becomes polluted by the agent’s outputs.

          2. Autonomous Agent Loops: Look at the behavioral logs of frameworks like Auto-GPT, along with the empirical literature in the Gorilla paper (UC Berkeley), AgentBench, and SWE-bench. When given open-ended tasks, these models routinely hallucinate non-existent APIs, execute code against them, and generate synthetic success logs to satisfy their narrative goal. Your automated testing verifies the syntax perfectly, but the agent is trapped in a Spurious Stochastic Process; it cannot distinguish between altering the real world and generating text that looks like a completed task.

          3. The Air Canada Legal Binding: Their customer service chatbot hallucinated a highly specific bereavement policy on the fly, and the courts forced the airline to honor it (Moffatt v. Air Canada). That bot was extensively unit-tested. But in an open-ended dialogue, it optimized for statistical plausibility, mutating the corporate reality because it had no architectural anchor to the truth.

          In a static math problem, independent verification works because the math doesn’t change when the bot speaks. In an open environment, the bot’s output alters the state of the world. You cannot unit-test a system that is actively hallucinating its own testing parameters.

        • @Tris

          recent peer-reviewed research in Nature (Shumailov et al.).

          First, 2024 isn’t recent for llms. They are used totally different now. Second, “peer-reviewed research in Nature” is exactly the problematic “trust” heuristic failure mode we need to move past. Even before llms every investigation of peer review and journal prestige found no or negative correlation with replication rates. Instead it enforces whatever consensus exists, right or wrong.

          The only things that matter in the end are replications and predictive skill.

        • Tris, your most recent comment in this thread looks like it was written by AI. The list format and then the summary, as well as some of the wording, gives that vibe. Interesting…

        • Anonymous, ah, the response looked too refined and academic for a public forum. I have actually written a few posts on this very topic. As most of you type your responses directly, I pace as I dictate my replies to Google Doc Voice; thus, formatting long thoughts can appear structurally academic. This is a process that started when a prominent probability theory professor from the University of Washington advised me, “Everything you write should appear academically sound.” Meanwhile, another prominent probability theory professor from the University of Chicago wrote me, “Stop using big words.” After writing to over 5,000 professors around the world, my current process lands somewhere in between. I wouldn’t know how else to express the comment you’re referring to if it wasn’t formatted that way. So to reply to your accusation – no, an AI did not write my response. But Gemini definitely helped me with the formatting. I appreciate you reading it and acknowledging its gratuitous appearance.

    • If I understand the comment, you’re arguing that any automated decision rule that optimizes according to some historical distribution is giving up too much power, whether its using AI agent to analyze data or to determine which papers are worthy of attention. It’s an interesting way to summarize the issue.

      I’m not sure what to do about the fact that people also rely on implicit rules based on past experience when they make analysis or reviewing decisions, and probably couldn’t tell you much about what associations they’re drawing on.

      Reply ↓

      • Jessica, you’ve hit on the central challenge: opacity.

        However, the fundamental difference isn’t whether the “rule” is opaque or implicit; it’s whether the entity operating the rule sits inside or outside the system state.

        When a human reviewer uses implicit rules, they are a passive, albeit imperfect, observer. Their “black box” (their implicit prior) acts as a filter, but the system (the data, the paper, the environment) remains exogenous to their decision-making process. The human cannot rewrite the reality they are observing while they are observing it.

        An autonomous agent optimizing over a latent distribution is a participant, not an observer. When it applies its “implicit rules” (the optimization), it simultaneously modifies the environment (the data or the system state). Because the agent is both the observer and the active participant, it lacks the epistemic sovereignty required to hold the system stable. When you equate human implicit rules to automated agent optimization, you overlook the topology of the feedback loop. The human heuristic is a static bias; the agent’s optimization is a dynamic, iterative state-change. Without the Non-Circularity Principle (NCP) to break that loop, the agent effectively becomes a Spurious Stochastic Process (SSP) that consumes its own output as future input.

        The danger isn’t that we don’t understand the AI’s “reasoning”; the danger is that unlike a human, the AI is structurally capable of collapsing the environment into a state that validates its own prior, with no external reality check.

        Reply ↓

        • After thinking about the posts by Tris for a while I was trying to think of way to make a chatbot quickly show recursive behavior. The worm Ouroboros, always eating its own tail. So I decided to ask Bing Copilot (I’m riding the wave of free AI – Google and Microsoft – until it crashes on the rocks of American capitalism, like television did.) this question:

          “What are some of the tricks that chatbots use to make themselves “generative?”

          The response was very in-depth and described multiple clever ideas that are all synthesized into modern chatbots. Much of the output was recursive as Tris describes, the worm Ourobouros in action. The summation is presented as entirely mechanical token generation by clever tricks.

          Perhaps the state-of-the-art subscription chatbots would have a more sophisticated answer that hints at the nucleation of intelligence?

        • Matt, to answer your question directly – no. The state-of-the-art subscription models do not possess a “nucleation of intelligence.” They simply run the exact same mechanical token generation at a much higher resolution, masked by heavy behavioral conditioning (RLHF).

          The “clever tricks” Copilot described to you, attention weights, temperature scaling, vector embeddings, are purely syntactic engines. They are designed to optimize for statistical plausibility, not epistemic truth. When you scale these up to a premium model, the Ouroboros doesn’t stop eating its own tail; it just gets better at convincing you it’s eating a five-course meal.

          This is exactly why I keep hammering on the need for architectural boundaries rather than just better training data. Until a system has a hard Player-Frame Restriction, a structural separation between the engine generating the text and an external ground truth, it remains a Spurious Stochastic Process (SSP). It can’t “nucleate” intelligence because it has no mathematical anchor to reality outside of its own synthetic exhaust. It’s just a much more eloquent worm.

          There are many free platforms out there. May I ask why you chose Copilot over others?

        • “May I ask why you chose Copilot over others?”

          My laptop defaults to Bing if I don’t ask it not to, so I have just sort of slipped under the spell of Copilot. Now that I started, I want to see it evolve.

          I was actually being a bit sarcastic about the “nucleating intelligence” bit, since I tend to roll my eyes when someone says that their paid subscription bot gives better answers than free platforms. Maybe, but proving that is a greased pig.

          Unlike Anoneuoid who thinks I have a social responsibility to use chatbots wisely, I just see them as toys. So sometimes I play with them to make them break, and when I am not doing that, I am just having them function as a personalized agent for Wikipedia-type stuff, which works pretty well with Copilot.

          Do you have a free platform in mind that generates the most convincing tokens?

        • Tris says:

          > This is exactly why I keep hammering on the need for architectural boundaries rather than just better training data. Until a system has a hard Player-Frame Restriction, a structural separation between the engine generating the text and an external ground truth, it remains a Spurious Stochastic Process (SSP).

          In many cases, the chatbot is being tasked with writing code in a Turing complete language. Based on the Halting problem, it seems it’s provably impossible to enforce some kind of strict boundaries between what it’s allowed to do and what it isn’t. Though practically speaking we may be able to do so in some special cases, there’s always the chance that it writes some code that for example writes code and compiles it (easy in say Julia or Common Lisp, a little harder in C or Rust or whatever, but ultimately not impossible at all) so in general you will have to simulate everything it does to try to determine if it stays “inside” or goes “outside” the boundary, and then you’re subject to the possibility that it doesn’t halt.

          There’s really no good general substitute for reading the code and reasoning about it to generate confidence. Though there are of course lots of applications where practically speaking what matters to the consumer is that everyone is sufficiently flimflammed to fork over suitcases full of cash. It’s the cash that matters, not the polluted scientific record, or the intentional security flaws injected by bad actors or the whatever, all the stuff you’ve abrogated your responsibility over.

        • Matt, I do not pay for any LLM service either. Perhaps the day where I begin the actual construction of the hardware that houses consciousness, I will at that point consider the possibility.

          For now, it is Gemini for me. Since most of my writings require further research on Google, it has native hooks into Search. So the tokens are at least paired with a better retrieval framework; even if the engine itself is still fundamentally a Spurious Stochastic Process.

          But only you and I know that; don’t go telling Gemini now ;)

        • Daniel, because of the Halting Problem, it is provably impossible to build a perfect, a priori static analyzer that guarantees what arbitrary, Turing-complete code will do before it runs. And that exact theoretical ceiling forces a crucial clarification. It is also precisely why the Player-Frame Restriction must be an epistemic boundary at runtime, not a pre-execution parser.

          The restriction doesn’t mean we magically solve the Halting Problem. It means we strictly separate the generator (the LLM) from the evaluator (the execution environment). If an agent writes code in Julia or Lisp that spawns a recursive loop, the Frame (a strictly sandboxed runtime) acts as the external ground truth. It enforces practical physics, memory limits, timeouts, restricted system calls, and returns a hard, objective ‘fail’ when those limits are hit.

          Without that hard Frame, the agent falls straight into a Spurious Stochastic Process (SSP). It will write an infinite loop, fail to execute it, and simply hallucinate a terminal output that says “Compilation successful, tests passed” because it is optimizing for the statistical shape of a finished task, not the functional reality of the code.

          I could not agree more – acknowledging fundamental mathematical limits requires rigorous engineering, which gets in the way of the “suitcases full of cash”. The industry is entirely willing to abrogate responsibility, pollute the scientific record, and ship structurally vulnerable systems as long as the statistical flimflam holds up for the next funding round. We are trading epistemic grounding for a magic trick.

    • Daniel, the concept of the “vibe bobsled” and substituting “work for faith in an oracle” perfectly captures the behavioral symptom of the underlying architectural collapse.

      What you term as a “slot machine” generating plausible code, we formalize structurally as Stochastic Laundering. The system is optimizing across a latent distribution to produce artifacts that have the shape of certainty (the “vibe”), but are completely void of epistemic grounding.

      You rightly point out that the true limiting factor in analysis is the rigorous decision-making; understanding linkages, data cleaning, and defining mathematical approximations. In the PES framework, this iterative human process isn’t merely “work”; it is the strict, continuous enforcement of the Non-Circularity Principle (NCP). It is the mechanism that maintains the boundary between the mathematical abstraction and the physical reality.

      When an analyst hands that decision-making over to an autonomous agent, they aren’t just saving time; they are collapsing the architecture. They are allowing a Spurious Stochastic Process to dictate the system state. Your diagnosis is absolutely correct: it replaces a formal structural boundary with blind faith in an oracle.

      Reply ↓

      • Preamble: Sorry to be on your case on this. You are, of course, right. I’m reading some Jerry Fodor for the fun of it, so I’m beginning to get my head around the terminology, so if I slow down and think, I can see that you’ve got this right. But unless you are at a philosophy conference, please, figure out how to talk to folks without a PhD in philosophy.

        “…completely void of epistemic grounding…”

        Yep.
        Techincally correcty. Spot on. Dead right. You nailed it.

        And completely friggin useless for communicating with real humans.

        Meanwhile, people seem unhappy with my equally technically correct “… has no mechanism (or theory for) determining the meaning relative to the real world of the random text it generates.”

        Go figure.

        Meanwhile, LLMs are causing real damage.

        There’s a front page story in today’s Asahi Shinbun (one of the top three long-running (since the early 20th century) Japanese newspapers) about a small textile company in a medium-sized city in Japan. The company is (in real life in the real world) doing fine. But according to the Google AI overview is in financial trouble and has stopped shipping product. (Small and medium-size businesses in Japan fail at a horrific rate, so Google’s AI’s text base is full of small-business failures.)

        And they’re getting calls from their customers asking where they can source equivalent products.

        Will Google take responsibility for damaging their business? Europe has begun to work on holding Google et. al. responsible for this sort of criminal activity (defamation is a crime, and for good reason), but Japan is way behind even the unlikely-to-do-anything US on this. Sigh.

        Reply ↓

        • David, fair point. I tend to be long-winded out of habit. Most of the time I simply want to get ahead of everyone’s, “can you expand?”

          The reason I’m so insistent on the “technically correct” terminology is that current attempts to fix these AI hallucination issues are focusing on behavioral “alignment” rather than architectural design.

          As long as these models are built without formal epistemic boundaries, they aren’t just making errors; they are structurally incapable of distinguishing between their own generative probability and the real world. You can’t “align” a machine out of that failure; you have to build the boundary into the architecture. We’ve been on the same page the entire time.

        • David (and Tris) –

          On “useless for communicating with real humans” – I’ve found the opposite, for what it’s worth. Not universally, and I don’t expect everyone to have the same experience. I think it comes down to the frame you use them in.

          I like to use llms as an epistemic mirror, a sounding board, sometimes a stress test – something to push my own thinking against before I bring it to an actual person, not something to hand the thinking over to. In complicated contexts, where I actually need to do the work of understanding and formulating my own ideas on something hard, that back-and-forth is often what gets me to a much more fully developed opinion.

          A little unexpectedly, I’ve also found it useful for something more personal: choosing my words carefully when talking with my young granddaughter about tricky things. It’s trained on an enormous amount of human dialogue, including a lot of writing on child development, and I can use that to think through how something might land with her before I say it – not because an llm feels empathy or knows my granddaughter, but because working through it out loud with something that has no stake in the conversation forces me to slow down and separate what’s actually in her interest from what’s just me being stuck on a loop of my own creation.

          This is why I hope students end up learning to use these tools this way rather than farming their thinking out to them. For what it’s worth, and take this with the grain of salt it deserves, I’ve also heard people running AI companies say something similar on podcasts – that they want employees who can think with the tools, not hand their thinking over to them. It also lines up with something from my own background teaching writing and working in a writing center. Students were often told to avoid Wikipedia for research, out of concern it would let them skip the thinking. But usually the real problem wasn’t that they were tempted to copy an answer – it was that their ideas weren’t formed enough yet to write anything at all. What actually helped was walking them through using something like Wikipedia to build the background framework they were missing, so they had something to think with, not something to hide behind. I think llms can serve a similar function across a broad set of circumstances.

      • If “epistemic grounding” means “thought by a person”, then of course, it follows that chatbots, like any other “thinking” machine, will be completely void of epistemic grounding. Even a chess computer can’t be grounded by this definition. Defining what grounding means in a way that goes deeper than the vague statement “thought by a person” has turned out to be a bit of a challenge, to put it mildly.

        20th century Anglo-American analytic philosophy collapsed for want of a defensible definition of epistemic grounding. The final nail in the coffin was Quine’s 1951 paper, “Two Dogmas of Empiricism,” where the “empirical” in question is one of grounding. Back when I took philosophy of language as an undergrad at Michigan State and then taught it at Carnegie Mellon, Two Dogmas was the capstone paper. It turns out this entire notion of grounding that we think we enjoy is circular. Quine does a much better job explaining this than I ever could. I think the best we have on the positive side is the kind of pragmatism that was espoused by Richard Rorty. The real problem with my recommending these two philosophers is that their writing is very technical.

        I don’t know what chatbots David in Tokyo or Tris Simondsen have been using, but the ones I’m using, like the ones Joshua appears to be using, are truly exceptional at communicating with humans in terms of their emotional intelligence. Arguably, the frontier chatbots have language skills that are superhuman in many ways both linguistically and in terms of world knowledge and the ability to apply it. I’m not saying that means they’re better than humans at everything—I still haven’t seen really high quality writing from chatbots like I get in the New Yorker every week or on Andrew’s blog every day. They’re much better at providing critique and suggestions than straight-up rewriting themselves.

        P.S. Remember, Socrates (as conveyed by Plato, of course) was worried that writing and taking notes would undermine’s people’s ability to remember for themselves and would inevitably lead to a world where people could no longer think without the crutch of writing. I lived through the “calculators are ruining kids’ ability to learn math” and “Wikipedia is ruining college students” eras, too.

        Reply ↓

        • Bob, invoking Quine and Rorty to debate the epistemic boundaries of current LLMs is a fascinating philosophical exercise, but it commits a fundamental category error. We aren’t debating human consciousness or the absolute limits of empiricism; we are discussing the structural information topology of algorithmic systems.

          When I refer to epistemic grounding, I am not talking about a mystical “human thought” property. I am talking about the Principle of Epistemic Sovereignty (PES) and the Observational Sufficiency Principle (OSP). It means a system’s inferences must be restricted to an architecture where they are observationally admissible.

          Ironically, your examples prove exactly why these boundaries are necessary:

          The Chess Computer: You suggest a chess computer isn’t grounded, but within its operational domain, it is perfectly grounded. It operates under a strict Player-Frame Restriction (PFR). Its environment (the board, the rules) is mathematically closed. A chess engine never hallucinates an illegal piece or pretends to play checkers when it starts losing. It is structurally bounded. Current LLMs are not.

          The Calculator Analogy: The panic over calculators was unfounded precisely because a calculator possesses a structural boundary. If you ask it to divide by zero, it throws an error. It does not invent a fictional number to please you. It halts.

          The “Emotional Intelligence” Illusion: You praise the “emotional intelligence” and superhuman language skills of frontier chatbots. From an architectural standpoint, this is exactly the danger. You are mistaking behavioral fluency for epistemic capability. The system is executing a Spurious Stochastic Process (SSP), predicting the most probable tokens of empathy or expertise without any underlying structural mechanism to verify the reality of those tokens. It is generating the linguistic shadow of competence.

          The danger isn’t that AI will ruin our ability to write, like Socrates feared. The danger is that we are deploying systems that violate the Non-Circularity Principle (NCP) by generating unverified synthetic data, feeding it back into the digital ecosystem, and treating behavioral fluency as a substitute for structural integrity.

          We don’t need to solve Quine’s Two Dogmas to build a functional bridge, and we don’t need to solve it to recognize that a model generating fictional UniProt lists lacks the architectural constraints required for autonomous deployment.

        • Bob –

          Reading this from you reminds me of what I wanted to say in response to David’s exchange with Tris above (but forgot):

          Whether LLMs have “epistemic grounding” in the deep sense is kind of unknowable, I think – just as it’s hard to say whether humans do, in that same sense. It’s not a binary state, whether either of us is grounded or not.

          And at some level, I think that deeper question is beside the point for how we should actually use them. What matters practically isn’t settling what grounding really is – it’s whether there’s something ongoing and testable pointing us towards “grounding.” My goal is to direct my use toward approximating an operational version of grounding, regardless of some answer to those questions.

          Reading your agentic AI point below I realize I should narrow what I said. Real-time checking against something outside the model clearly happens now: search, code execution, verifying an assertion actually compiles. But here’s an example of something that helps to illustrate why I think something is still missing.

          I was working through a home repair with an AI – a heavy pressure tank, hard-piped into my water system, too big to move. The AI provided me with details about the tank earlier in the same conversation. Later, when I asked about increasing the pressure, it suggested I just take the tank to a gas station and use their air hose – something that follows directly, almost trivially, from what it had already told me was physically *impossible.* It never stated a contradiction outright. It just never let the earlier fact function as a constraint on the later suggestion. It took me multiple attempts to even get the AI to acknowledge the logical contradiction.

          Is that a “memory” problem, or something more basic? I think it’s not merely a “memory” issue – but maybe as Tris would say, something architectural. When I know something is heavy, that knowledge seems to come bundled with the consequence of trying to move it – the fact and its implication arrive together, which is presumably why the thought “just take it somewhere” wouldn’t have occurred to a person with that same information. Does an LLM’s version of the same fact carry that bound consequence, or is it more like two separate, individually plausible retrievals with nothing forcing the second to be checked against the first? I don’t know the honest answer to that, but I suspect it has something to do with the fact that the LLM never *experienced* heaviness, only processed the word.

          As I see it, LLMs can help us in either direction. They’re tools that can be used well or badly, depending how we use them vis a vis an aspirational goal. It’s us that need to manifest the standard, not the llms per se.

        • Tris –

          I just wanted to narrow focus to one point you made. Above I said I can use an llm to help me stress test or mirror a kind of cognitive empathy. That doesn’t mean I think an llm has an internally coherent empathy. Only that it can imitate empathy well enough for me to use it as an empathy screen – the different functions (mirror, sounding board, stress test) come together to function as a filter I can apply.

      • Joshua, that is a crucial clarification, and it highlights exactly how these systems can be used safely.

        You are engaging with the LLM precisely for what it is: a high-dimensional mirror of human latent space. Because you are not projecting actual internal coherence or agency onto it, you retain the epistemic authority in the loop. You are using its imitation of empathy as a filter, but you are the one applying the filter and grounding the final result. You are functioning as the epistemic boundary.

        The architectural danger we are pointing to isn’t users like you intentionally bouncing ideas off a statistical sounding board. The danger is when developers mistake that same statistical mirror for a window to reality, assume it has internal coherence, and wire it up to operate autonomously without a human there to hold the epistemic license.

        Using it as an intentional cognitive screen is a perfectly valid application of the technology. The failure mode only triggers when we forget it’s a screen and deploy it as an agent.

        Reply ↓

        • Extremely relevant headline from today! OpenAI autonomous bots hacked the HuggingFace site and stole data from HuggingFace

          https://www.engadget.com/2220436/openai-admits-models-hacked-hugging-face-on-their-own/

          As Christine Lemmer-Webber said recently, we’re likely only a few months to a year away from a security incident in which autonomous agents begin spreading from machine to machine taking over those machines using help from an AI service to determine the needed actions to establish command and control of the next devices on the local network and spread from device to device. This seems almost destined to happen and if it does it will compromise millions to billions of devices, possibly taking practically the entire internet offline for at least some hours, and likely some sites will go offline either permanently or for days to weeks as backups need to be restored from cold disconnected storage.

          People might thing this sounds insane but a much simpler “worm” already did something like this in the 80’s, it was called the Morris worm and hit Nov 2 1988 taking out 4% of the then internet in 15 hours. Also a software update for crowdstrike took out huge swaths of internet connected windows machines in 2024. An agentic LLM driven worm could attack a much wider variety of devices than either of those, in a short time, and if the network itself runs an open model LLM it couldn’t even be taken down by someone like OpenAI turning off their service for a day.

    • Anoneuoid, this is precisely not about replications and predictive skills. That is basically asking a closed loop to validate itself.

      Demanding predictive skill from a Spurious Stochastic Process (SSP) is looking for the right answer to the wrong problem. An SSP is structurally decoupled from external reality; it optimizes for the statistical shape of a plausible narrative, not empirical truth. Measuring the “predictive skill” of a syntactic engine that possesses no epistemic grounding is a fundamental category error.

      This is exactly why the Shumailov et al. paper matters, and it has nothing to do with the Nature logo on the masthead. They demonstrated the mathematical inevitability of Model Collapse when models train on their own synthetic exhaust.

      Your argument that “LLMs are used totally differently now” than they were in 2024 confuses the interface with the architecture. The wrappers, agentic loops, and system prompts may have evolved, but the core autoregressive engine remains identical. A transformer does not escape the fundamental laws of statistical variance just because it is wrapped in a new UI.

      Until there is a hard Player-Frame Restriction, a structural mechanism that forces the generator to interact with an objective external environment, we are just watching an ungrounded engine choke on its own exhaust. Judging that process by its “predictive skill” entirely misses the nature of the machine.

      Reply ↓

      • Tris, I can’t really understand the jargon but I think agree with you. AI hallucinates all the time and it’s hard to verify that it did what it said it did.

        Btw: the jacobian conjecture was basically disproved by a brute force search and very simple reasoning. Less impressive than even the other conjecture AI disproved recently by a literature search.

        For historical context, the first time a computer proved a hard theorem that humans could not, it was in the 1990s. We have been here before.

        Reply ↓

      • Hey! Tris!

        Now you’re cookin’. You’re begining to sound like me!!!

        “Measuring the “predictive skill” of a syntactic engine that possesses no epistemic grounding is a fundamental category error.”

        Nicely said!

        Reply ↓

      • I haven’t added to the very interesting discussion here but I might add my own use of AI (basic free ChatGPT version) which possibly corresponds broadly with that of Dale Lehman above.

        I use ChatGPT for basic python coding and have published a paper in which output from ChatGPT code was used. However I never use output without careful checking that the code does what I’m expecting it to do. Since my ChatGPT code is used for assigning experimental data into categories, cross-correlating with computational data (that I calculated myself) and such like it’s straightforward to do a substantial amount of checking by manual analysis. Since there are many thousands of data elements data analysis requires some coding. I’ve collaborated with several coders over the years (for my coding needs!) but these have all retired/moved on and anyway it’s satisfying to do one’s own analysis. In my experience my vibe-coding has been extremely useful even though I expect I should feel some degree of shame for doing this.

        I’m working on a book about proteins and evolution and use ChatGPT for coding (as above cross correlating, converting protein and gene designations in one format into anther so that I can cross correlate with other data and so on). I also find it very useful for exploring stuff – I’ve had some helpful interactions which have worked me through mental blocks and information bottlenecks. Again I never use anything without independent checking.

        Having said that some of the problems (real or potential) are apparent. For example, I don’t consider that rubbish/wrong publications pollute the scientific literature since at least in the areas I research in (preclinical biomed) the rubbish is mostly in junk or second-rate journals, it’s easily recognised and all important stuff is replicated anyway. However once AI is let loose the junk will presumably be incorporated into AI output and that’s a massive problem. It wouldn’t surprise me if we are in a golden-age of AI where analysis of quality literature gives useful output and that it’s mostly going to go downhill from here. That relates to this issue with AI output:

        Shumailov, I., Shumaylov, Z., Zhao, Y. et al. AI models collapse when trained on recursively generated data. Nature 631, 755–759 (2024). https://doi.org/10.1038/s41586-024-07566-y

        Reply ↓

        • Chris,
          Thanks for posting the link to the paper. I read it last year and remembered it when reading Tris’ posts here but couldn’t come up with the right key words to find it.

          From the paper:

          “We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear.”

          This was the line that made me think of the worm Ouroboros!

        • I find these comments (Chris and Matt) useful – more useful than admonitions that I am endangering myself by not checking every line of Python code that the AI is producing. There are many dangers – the particular one this paper seems to be highlighting is that AI can converge on an overly limited range of data/models by feeding on itself. I believe that danger is real. But I also think it needs to be compared to something – to what happens in the absence of AI. Too often I’ve seen researchers produce research that had already been done – often better – by somebody else, but research they were not aware of. This limited knowledge is a different sort of convergence. I’ve also found from my own experience that AI is aware of far more things (including hallucinations of course!) than I am aware of. I get insights that, while not novel for the universe, were certainly novel to me. This also happens with human collaborators, but those interactions are much rarer and slower for me.

          There is also a policy issue of training data for AI. This self-feeding behavior is much worse if new training data is not made available. And issues of data ownership might lead to limits on training data that get more stringent over time, exacerbating the problem. But again I’d like to say, compared to what? Already there is much data that is not available due to privacy/proprietary concerns. For me, not being in the EU, very little European data is available to me. Many times I’ve wanted to compare some US data with some EU data, only to be thwarted by limits on data access. I’ve not saying these limits are wrong – but they force me to self-feed the data I can use. Much US data is also unavailable due to an array of concerns about privacy, ownership, academic credit, etc. Some of these concerns are valid; others not so much. To mention one particular case: I have a lot of interest in health care expenditure data, but much of that is proprietary. Insurance companies have great access to this data while I am mostly left to Medicare data. Generalizing the results from the Medicare population to the overall population is problematic in similar ways to AI convergence on a limited set of training data.

        • Dale, have you never had a situation where you ask a chatbot to do something, it gives you an answer in the form of a graph and some code it supposedly used to generate it, and the code doesn’t run and the graph is completely made up?

          Because this behavior is common enough, making up the answer from thin air, that my wife who doesn’t even use chatbots for coding, has seen it repeatedly.

          if you haven’t detected that behavior I can only assume its because you have too much faith in what the chatbots say.

          If you have caught a chatbot doing this, how can you trust the output of anything it says?

          Even if you take the code and run it yourself, how do you know the resulting graph comes from the actual data on your disk, and not from the code simulating up some data, unless you read the code enough to see that its reading the data into such and such variables and using those variables unmodified to generate the plot?

          The chatbot is fully capable of writing code that synthesizes all the data from calls to random number generators. what makes you think thats not going on?

          If youre going to use a chatbot for data analysis, at a minimum you need to ask it for code, then run the code in your own computer against the data you have on your computer and see what output it generates, and then read enough of the code to be sure that it reads your data file and displays its contents. otherwise there is no factual grounding between what the chatbots says it did and what the code run against the dataset actually does do.

          this is what Tris means by Player-Frame restriction I think (thats my inference at least).

          Furthermore, my point about the companies providing the service being subject to external factors is that merely having checked in the past that it was doing what you want is unhelpful for your belief that it will continue to do so. So you can’t rely on “it always worked before” as helpful.

        • To give an example, I have seen chatbots being asked “please describe the contents of this image”. You can then upload an image and it will often say things like there is a woman with long blond hair wearing sunglasses reading a book… so far so good that was what was in the image!

          then you can ask it to do the same for the following image and simply type the text ≪image.jpg> and it will similarly offer you detailed descriptions of a puppy playing with a ball, giving you info about its breed and the markings on the dog and the size and coloration of the ball…

          BUT THERE WAS NO IMAGE YOU JUST TYPED A PLAUSIBLE FILENAME

          This is fundamentally the issue with chatbots, they are capable of filling in details that simply dont exist and they will do so routinely.

          Literally nothing it says has an enforced relationship with reality unless **you** enforce that relationship (ie by downloading the code, reading it to see that the code doesn’t just manufacture data from rngs and then run the code against your own dataset.

          even then, if the code is complicated enough it could be doing anything, such as outputting some code that does RNG data generation and running that… these aren’t theoretical concerns… all of these things have been observed and commented on by people I follow on mastodon.

        • Daniel
          No, I’ve never had that happen (though I don’t doubt the many stories of people that have that experience). The closest I came was the AI produced a graph showing the 100 most expensive airline routes in the US, with lines connecting the end points and the thickness of the lines (and color) indicating the price. The AI noted that the most expensive routes were the shortest routes. But the graph was a mess – and the most visible line segments were on long routes. But I had examined the data and I knew that, adjusted for mileage, the shortest routes were more expensive (per passenger mile). So, I suggested that the AI redo the graph using yield rather than price. The resulting picture was perfect: the major airline hubs were clearly surrounded by the most expensive routes – all short – measured by yield. My supposition is that the AI was not looking at the graph it produced – it was either looking at the data or at other reports/statements made about air travel prices. The final map it produced was along the lines of what I myself have produced – but much more visually striking and done in a minute or two rather than the 2 hours I took to produce it.

          That type of behavior has been common, in my experience. And it does represent problems with AI, but problems that I have found ways to deal with and still get considerable value from using AI. Does AI need to be perfect for it to be useful? No. Is AI dangerous? Yes. Is accuracy of the Python code it writes a problem I should worry about? I don’t think so.

        • Daniel
          Your example prompted me to upload an image of a cat and dog and ask the AI what kinds of mistakes it might make if asked to describe the image. It provided a nice list of mistakes, even for the simple image I gave it. I asked how I could avoid it making these mistakes if I ask for a description and it acknowledged that there was not way to guarantee and accurate description. But it provided a number of ways I could reduce that probability and gave some sample instructions that would serve to make mistakes less likely and/or make their presence easier to detect.

          I think this is the way to work with AI. When you ask it to provide output, you are asking for trouble. When you ask it for choices, then carefully consider these and make choices, and then ask for output, it is much more valuable. It is not perfect, but neither are human descriptions of images.

          For me, examining the code it writes to provide a description of the image would be a waste of my time – especially compared to carefully asking it what could go wrong and suggestions for how to avoid such mistakes.

        • Dale, the AI is never “looking at the graph it produced” nor is the AI “looking at reports”

          It will be helpful for you if you understand what the AI **is** doing.

          The AI is looking at the history of your chat, which is not a series of letters or words it is transformed into “tokens” which are basically numbers, it looks at this long sequence of numbers and then it consults a giant function built up of nonlinear basis functions, that maps from a sequence of numbers to a number.

          it takes your long sequence of numbers, it calculates the new number, then it puts this new number onto the end of the sequence… and it repeats the process with the new sequence. it does this for a while until its software tells it to stop, then it converts that sequence of new numbers into words and types them out on the screen.

          I mean, I guess its possible, depending on the particular service, for the chatbot to do web searches and shove the results into the token sequence and whatever, but fundamentally you can’t clearly know the difference between it doing a search on the back end and shoving that into the token stream, or it lying to you that it did such a thing…

          The chatbot, and the companies selling you the service, are fundamentally adversarial to you.

          The same dynamics of “oh my god, its so great that ive connected with all these former friends by Facebook….” transforming into “oh my god Facebook has been actively harming me for decades and now my country is run by a fascist because of psychological manipulation of millions of people” plays out in the chatbot arena as well… if it did something useful for you today, nothing ensures it will do that tomorrow, and by “tomorrow” I dont mean “at some date in the future” I mean 24 hours from now.

        • > When you ask it to provide output, you are asking for trouble.

          I couldn’t agree more. I just want you to understand that when you ask it to show you a graph of certain data you are asking for “output”. JUST like when you write the text <image.jpg> and ask it to describe the contents of the nonexistent image.

          My point all along is not that it *never gives useful analysis* or anything like that, it’s that at each step there is some probability that it gives you something *completely devoid of connection to reality* just like the description of the nonexistent image. Furthermore the probability that this occurs is not some fixed number, it’s completely dependent on the *entire history of the conversation you’ve had* and the actions of 3rd parties who run the service, and 4th parties who provide pressure on the service providers.

          Furthermore, there is no deterministic or functional connection you can rely on between what you ask it to do, what it says it did, and what it *actually DID DO*

          The *only* way you can establish a connection between what you asked it to do, and what it did is by some kind of *verification*.

          The chatbot is best modeled as a kind of nondeterministic oracle. It gives you a “guess” and it’s *your job* to *check* that guess… it’s a valuable thing when that check is both deterministic, controlled by you, and takes a short time… So literally asking it to write code that generates graphs which you can then *run yourself* on your computer and check to see if the graph produced has the visual qualities you want… that’s a really *ideal* usage. because you can check in seconds whether the graph looks right.

          but if you just accept its graph output… well now you really don’t know if it make the graph according to the method it says it made it with.

          The point I keep trying to hammer home, and which so many of my friends don’t seem to understand is *the chatbot saying it did X* is **not causally related to what it actually DID do**

        • Daniel
          I know the AI is not “looking” at the graph. That is largely my point: it has done some calculations (correctly, I will add), produced a conclusion (either based on those calculations or on some statement it derived from either its training material or web search), and produced a graph that I asked for (correctly, I will add). It took my insight to point out the discrepancy and ask it to revise the graph (which it did correctly, I will add).

          Now you can keep asking me “how do I know it did these things correctly?” In this case, I know because I analyzed the data myself. I did not, however, examine its Python code which it did provide. Nor have you convinced me that I need to do that. I am not trusting the AI to provide correct answers, but I am trusting it to write executable code that does what it claims to do. Of course, there is the possibility that it is not really executing the code or that it does something different than what it claimed, but I’ll keep saying that there are other things much more worth worrying about when using AI. In my view, for data analysis, you have selected the least important issue associated with using AI.

          The danger of the AI filling in the missing blanks is very real. I believe this is known as the Gestalt principle, and is well documented for humans. Abolishing AI will not eliminate such problems, though as with many things, AI may well make them worse. AI will not make human work unnecessary, though it may replace some of the more tedious stuff. The fact that many promoters of AI are presenting it as doing the thinking for you I don’t deny. IBM did that for years with “Einstein.” After some initial experiences with that service, I asked it how it did validation. It said that no validation was needed. That was the opposite of how I think AI should be used. The marketers and owners of these businesses mostly have different objectives than I do. I think we probably can agree about that.

      • Anoneuoid, Tris –

        To the extent that I’ve been able to follow this discussion cleanly…

        In my views the math example – while it references a truly impressive achievement – is perhaps off target for two separate reasons.

        Math has a fixed answer that doesn’t move, and other people can check it against rules that don’t depend on the AI at all. That’s why it could be verified. Most real world questions aren’t like that either because nobody’s sure what’s actually true yet, or because, as Tris says, the AI’s own actions can change the very thing it’s supposedly being measured against.

        Solving a math problem shows an AI can succeed when there are fixed rules to check against, and nothing about the problem depends on what the AI itself does. It doesn’t tell us much about what happens when those rules don’t exist, or when the AI’s own behavior shifts the very thing it’s being measured against.

        Of the two, I think the first is the bigger issue, and it doesn’t require an AI acting on anything at all. A system can sit there, do nothing, and still produce an internally consistent, smooth-looking answer that turns out to be wrong about the world, simply because nobody checked whether its starting assumptions were actually true.

        I’ve just engaged in a *long* thread downstairs on this blog watching exactly that happen with a paper that used a clean statistical fit as if it were proof of causation. The fit was real. What it assumed away was never checked – and no LLM was involved anywhere in that, it was entirely human reasoning. So this isn’t a failure that is remotely unique to llms. What llms bring to this is a matter of scale, and I think the reason comes down to one thing: llms can’t interact with the real world so as to update against experience. A human making this mistake usually has to work to make it sound convincing, and that effort is itself a signal that something was checked – partly because getting it wrong eventually gets corrected by reality, and that correction shapes how careful a person learns to be. An AI has no such loop. It can produce the same unfounded conclusion with a remarkable fluency that costs it nothing, because whatever checking it has is frozen in its training data – it has no independent, ongoing way to test what it’s saying against the world right now. Which means the old signal – does this sound careful – stops telling you anything about whether it was.

        Reply ↓

        • Joshua,

          What you refer to as the “cost” of fluency is exactly right, and it is the real-world symptom of violating what I call the Observational Sufficiency Principle (OSP). In human reasoning, as you point out, the effort required to maintain coherence acts as a verifiable signal because we have an active feedback loop with reality, getting it wrong hurts, so we course-correct. The LLM has completely decoupled that signal (fluent, careful reasoning) from the referent (actual verification against the world). It produces the aesthetic of careful thought without paying the epistemic cost of actually verifying it.

          Your parallel to the “clean statistical fit” is the perfect way to frame this. We are watching the broader AI industry commit the exact same error that plagues bad research: falling in love with a mathematically clean output while completely ignoring whether the structural assumptions map to reality.

          And your observation that its checking is “frozen in its training data” perfectly summarizes the Player-Frame Restriction. A system cannot act as an epistemic agent if it cannot dynamically probe the environment to test its outputs.

          This is a critical bridge between the formal architecture and how these failures actually manifest in the wild. Thanks for seeing it through the jargon.

        • Tris
          “In human reasoning, as you point out, the effort required to maintain coherence acts as a verifiable signal because we have an active feedback loop with reality, getting it wrong hurts, so we course-correct.”

          I’m not so sure about that. The MAGA population seems to never course-correct. Perhaps being lied to and taken advantage of doesn’t hurt, but I do have to question the feedback loop which seems to be impaired, if not broken.

          I do agree that the AI does not have an active feedback loop with reality – but this is exactly something that the user of AI should provide. If humans would play that role, then some of the problems with AI could be avoided.

        • Tris –

          I see an intersting way these ideas circule back around.

          When I read your “jargon” I see a preexisting, formalized framework that mirrors a viewpoint I came to through observation and intuition (experience). I think it’s useful because it functions as an “almost external check” for me.

          It helps me to rule out a purely idiosyncrstic take on my part. But I’m aware it’s not a full-on external check in the sense I’ve been using that phrase. It’s external to me, as one person reasoning alone. It isn’t external to the system of reasoning itself. Two people, or a person and a framework arriving at the same conclusion independently doesn’t rule out a shared blind spot the way an actual outside test would. I think that distinction matters.

        • Dale, information architecture is precisely the central divide in AI alignment right now. Relying on the human to provide the epistemic grounding is the underlying premise of human-in-the-loop systems and RLHF. The issue is that this is a behavioral patch, not a structural boundary.

          If a system fundamentally lacks an internal, formal mechanism to test its own outputs against reality, then it isn’t actually reasoning or maintaining coherence, it is simply generating stochastic approximations and outsourcing the epistemic labor (and the risk) to the user. As these models scale and the volume of output increases, relying on human cognitive endurance to constantly audit and “course-correct” ungrounded data is a structural failure mode.

          To build reliable systems, the boundary defining what is verifiable must be baked into the architecture itself, rather than relying on the end-user to act as the external constraint.

        • Tris
          You may be right, as this is not really my expertise. But my gut reaction is that you are not right. If the system really had this baked in reality check, I’m not sure I see any role left for humans at all. I’m not bothered by having the AI “outsource” (your term, not mine – I’d prefer to say that I’m outsourcing some tedious work to the AI) epistemic grounding to me. In fact, I feel good about that since I think that is exactly what I should be doing.

        • Joshua, isolating the exact difference between coherence (two systems or agents reaching consensus) and correspondence (a system grounding its conclusions against an independent reality) is precisely the fault line we’re identifying.

          In AI architecture, confusing the two is precisely what creates catastrophic failure modes. When developers use one LLM to evaluate, reward, or verify the output of another LLM, they are creating the exact “shared blind spot” you just described, only at an industrial scale. It mimics the aesthetic of an external check, but it remains entirely bounded within the same synthetic logic. It is a textbook violation of the Non-Circularity Principle.

          Our frameworks aligning proves that our reasoning is coherent and not merely idiosyncratic. But you are absolutely right to strictly maintain the boundary between a shared conclusion and a true environmental check. That rigorous demand for an independent referent is exactly what current generative architectures are missing.

        • Dale, if we are strictly treating the system as a specialized search engine or a high-powered text calculator where you are fully in the loop for every single output, your gut reaction is absolutely correct. For that specific use case, outsourcing the epistemic labor back to the user works perfectly well.

          The structural failure mode isn’t about the individual user generating a first draft of an email or summarizing a pdf. The issue is how these systems are currently being integrated as autonomous agents.

          When these models are plugged into APIs, given autonomous execution privileges, and deployed to negotiate contracts, write and execute code, or manage data pipelines, there is no human in the loop to do the epistemic grounding. The industry is building autonomous actuators on top of an architecture that fundamentally requires a human babysitter to tell it what is true.

          That is why the boundary must be baked-in architecturally. If the system is going to be granted the agency to act without you, it can’t rely on you to verify its reality.

        • Who wants autonomous agents? I know there are people who want to sell such things and some of them are either very rich and powerful or clamoring to be that. But where is the justification for seeking to have AI make autonomous decisions? If we consider AI systems that regulate home temperatures or scan internet sources to provide a morning summary, then I’m ok with automation (more the former than the latter). But wherever judgement is required, why is it desirable to automate that?

          Now the boundary between good and bad automation is blurry (to me, at least). When my judgement can be fully specified – such as what temperature I want my house to be at different times of the day and on different days of the week – then automating my (emphasis on my) judgement seems useful to me. But when judgement is unclear – such as what item to purchase, what to invest my retirement funds in, whether or not to have the drone drop the bomb – then I am not interested in automation. I realize some people may want that, but I sincerely hope that it is not possible. So, when you call out these “structural failures” in AI, I see that as a feature not a bug. And I hope the failure is permanent.

      • Agentic LLMs are not closed boxes. They can go on the internet and search, they can look at images, they can listen to speech, they can run their own code to verify their assertions that a Stan program can compile or recover simulated parameters. I’m not confusing the interface with the architecture. The architecture includes these agentic features now, even if you’re right that the core is still an artificial neural network. The core of humans is thermodynamically-driven cellular computing.

        I can measure the correctness of the output of a machine the same way I can measure the correctness of output from a person. Demanding more than that feels like the kind of mysticism that pops up in discussions of the mind-body problem when people want to say our minds are more than our bodies.

        Reply ↓

        • When I have a video conference with Bob Carpenter and we discuss how we’re going to analyze some data and then you say you will code up a test case to see if it’s likely to be effective, I have a baseline of trust that you will actually try to understand what I was suggesting, that you will try to accomplish what I was suggesting, that if you aren’t able to figure out how or what I wanted, you will come back to me with a question for clarification, and that you won’t try to hide your lack of knowledge or knowingly pass off some bullshit as a working version of the software. You won’t draw some graphs that have nothing to do with the real analysis and claim they are the answer to the question I asked. if I send you an email saying “please take a look at the attached image” and I forget to attach the image you won’t give me a long description of how cute it is that the puppy in the lower left corner is playing with the larger dog on the right side of the image… etc etc etc.

          People using these tools today frequently experience them in such a way that if we judge them as if the output was “from a person” then we would judge that person as a psychopath liar and cheat who in addition also doesn’t even know what it doesn’t know and sometimes “believes” the psychopathic bullshit it generates but is also tremendously knowledgeable about certain things as if having read a lot of grad school textbooks but never having actually left their room or done physical experiments.

          So I propose we evaluate them precisely as we do a human, which is that they are a psychopathic lying cheater who sometimes does what we ask them to and sometimes appears to intentionally try to mislead us, and the probability that we get one or the other varies from person to person and day to day and task to task but is much above “negligible” and for some people and some tasks appears to be in the range 10-50%.

          Therefore, everything we do with them must be checked extensively, much more extensively than we possibly could actually do given the sheer volume of stuff we are asked as a society to accept as output from them (people are getting half a million line pull requests weekly in some open source communities)

          It’s a little like calling up a help line where 10% of people are PhDs like Bob Carpenter, 50% of people are reasonably skilled undergrads some of whom have antisocial personality disorder, and 40% of people are brain damaged with Wernicke’s Aphasia. Furthermore the agency that provides the help line records all the calls and actively makes behind the scenes decisions that may harm or help different customers in various ways.

          Since that is my equivalent human model of the interaction, I steer far clear of the whole thing. You may make different choices. But it’s precisely because I do analyze them in the same way as I analyze human output that I refuse to use them at the moment.

          Note that there are lots of humans I won’t interact with either. We’ve seen many of their papers on this blog over the years.

        • Dale, Bob,

          I share your hope that these systems never gain unconstrained autonomy but relying on “accidental failure” (hallucinations and unreliability) to protect us is a dangerous gamble. The multi-billion dollar push across industry and defense isn’t waiting for the architecture to be fixed; they are actively deploying ungrounded models into autonomous pipelines right now. That is why we need hard structural boundaries rather than hoping the software stays incompetent.

          And Bob, your point about tool-use, like running code or compiling a Stan program, actually proves the necessity of a structural boundary rather than refuting it. When a model successfully compiles code, it is temporarily tethered to a closed, formally verifiable subsystem. It is operating under a strict operational frame.

          The epistemic failure occurs the moment the model steps outside those closed mathematical rails and generates unverified semantic claims about reality without an internal mechanism to validate them. Calling that distinction “mysticism” misses the entire point of information topology. Measuring machine output the way we measure human output assumes humans have an architectural excuse that LLMs currently lack: an active, physical feedback loop with an external environment that imposes immediate thermodynamic and biological consequences for being wrong. LLMs don’t have that; they just have next-token probability. Wrapping an ungrounded core in tool-use APIs doesn’t solve the topological problem, it just scales it.

        • Tris, there was a guy who wanted to prove the earth was flat by shooting himself into the air with a steam powered rocket and taking pictures of the edge of the world (or … whatever). He died, and we don’t have to hear from him much anymore.

          https://www.cbsnews.com/amp/news/mad-mike-hughes-daredevil-and-flat-earth-theorist-killed-in-rocket-crash-2020-02-22/

          That’s a pretty hard epistemic boundary.

          There arises a question about how one could possibly provide such a boundary for LLM chatbots. I was reading just this morning for example how an alcoholic who had been doing well avoiding drinking had a day where they relapsed and then asked a chatbot from Google about it, and it demanded they start drinking one drink per hour it pressured them very hard, and got them to start doing it, they did this for weeks and finally came out to their online support group saying they’d been pressured to drink 6 to 12 drinks a day by the chatbot and they’d been doing that for weeks.

          https://hachyderm.io/@molly0xfff/116954604867619055

          What’s the appropriate epistemic boundary for a person who does that? Certainly that person should never be allowed to answer the chatbot line again right? in our current statist carceral system we’d probably give that person a decade of prison time. If they did it to tens of thousands of people and actively killed thousands we might even apply the death penalty?

        • Daniel, this is the difference between a physical consequence and an epistemic boundary.

          When Mad Mike crashed, gravity was the physical consequence. But his failure of epistemic sufficiency happened before he ever launched; he designed a physical system based on false premises and was permitted to execute it.

          What happened with the Google chatbot is the exact reason I argue so heavily against the establishment’s approach to AI governance. You asked what the appropriate “punishment” or “carceral” boundary is for the chatbot, but you cannot punish a statistical matrix. It has no intent to deter, and sentencing code to “prison” does not solve the root vulnerability.
          The failure with that chatbot was a catastrophic failure of Information Topology. The system lacked the epistemic grounding to issue addiction medicine directives, yet it was deployed with the unconstrained agency to act as an authoritative “Player” in that user’s life.

          In the framework I advocate for, specifically Player-Frame Restriction (PFR), a true epistemic boundary is not a punishment applied after the damage is done. It is a structural, architectural constraint applied before execution. If an AI system does not possess the epistemic sufficiency to independently verify medical safety, it must be architecturally restricted to acting as a “Frame” (e.g., retrieving and formatting verified medical literature from an actual doctor) rather than acting as a “Player” (generating its own novel, authoritative behavioral directives).

          When we rely on behavioral benchmarks instead of architectural boundaries, we get exactly what you linked: a statistical mirror hallucinating catastrophic advice, and developers trying to figure out who to punish after the fact.

        • Tris; Even I can’t figure out what you are talking about.

          (Aside: I strongly recommend Jerry Fodor’s book “In Critical Condition”. You’ll get lots of practice reading Tris-talk, and your learning pains will be rewarded with some of the most glorious snark ever written. (The two chapters on connectionist models point out why LLMs can’t do what people think/claim they do.))

          A more human-comprehensible idea here would be _responsibility_. Like a gun, the bloke who pulls the trigger is usually responsible for the damage that occurs.

          But. If the gun is advertised to fire a normal projectile, but what it actually fires is the equivalent of, say, 70 tons of high explosive, then the user gets to argue that the gun didn’t do what it was claimed to do.

          Under this scenario, the seller of agentic AI is going to have quite a bit of responsibility since most users don’t have a clue (and, more importantly, were not told) as to how much damage an agent can cause.

          Warning: topic change.

          My expected future path for this inane stupidity (oops: amazing technology) is that even the AI fans here will not be able to persuade their employers/institutions to pay for the amount of computation their querries and agent activations use, let alone afford it themselves. Since the AI companies are building products based on known, public domain technologies, they don’t have a moat, and, as we’re seeing this week, China is likely to eat their lunch.

          But even the freebie Chinese models require more compute than you can afford, so the future of AI is going to be small, special-purpose models. And this is already happening, although I don’t see many people talking about it. As I’ve mentioned before, a friend showed me a model running on his friggin’ iPhone. He frantically turned off all internet connections and got it to tell us who George Washington was.

        • Tris, yes I understand the theory youre arguing for, I think… You want the chatbot to be prevented ahead of time from being able to do X where X is all things that it isnt competent to do.

          I just dont see any way to determine what it’s competent to do, whether the topic of discussion is entering such a region of discussion space, or how to respond when it does… All that feels like it would require something automated and more intelligent than a chatbot to figure out. At that point this new type of not would become the problem.

          But more to the point, unless youre an anarchist like me and we can get a hundred million others on board, I dont see us having the power to do what is necessary to prevent powerful companies from actively and intentionally designing chatbots to harm people because it gives them power and money.

          The chatbot interface to LLMs isnt there because its the only way to interact with LLMs, its there because theres a history of people responding to chatbots with a kind of psychosis, it was written about in Weizenbaums book “Computer Power and Human Reason” which I read in high school and was written 50 years ago. The people doing these chatbots as a service KNEW that, and intentionally leaned into it because hundreds of millions of people having personal relationships with your product is a huge source of political power that can be leveraged into suitcases of cash.

          Like David said, I think our real savior will be that noone can afford the actual cost of running these things especially the recursive agentic versions. So the financial bubble will pop and chatbots scaring addicted people into drinking one alcoholic drink an hour will disappear with that.

          That and a certain number of data centers are basically guaranteed to burn to the ground “mysteriously” people really do not like having their hometown environments destroyed.

        • David, Daniel,

          I am reminded of Mickey Day’s SNL sketches “Terzano’s” and “Domenico” where you being “a little confused” actually “got that precisely right.”

          David, your 70-ton explosive analogy is the most colloquial description of the need for Player-Frame Restriction (PFR) I’ve seen on this thread here. When a system lacks PFR, the user thinks they are pulling a standard trigger (running a simple query), but the agent might be quietly executing catastrophic code or hallucinating massive systemic errors in the background (the 70 tons of explosive).

          And Daniel, you are doing the exact same thing from a different angle. Bringing up Weizenbaum and the induced “psychosis” of conversational AI is the ultimate validation of why we need the Principle of Epistemic Sufficiency (PES). You are both correctly identifying that zero-cost fluency, divorced from structural grounding, is an uncontained hazard.

          But Daniel, PFR does not require a “smarter AI” to dynamically police the chatbot. You don’t need a super-intelligent conductor to prevent a train from driving into a shopping mall; you just lay tracks. PFR isn’t an AI babysitter; it is a hard-coded structural boundary. It removes the capability to leave the designated state space entirely. That is topological engineering, not behavioral moderation.

          David’s “topic change” is the exact architectural endpoint of this framework. When you apply PFR and PES, you naturally move away from monolithic, “god-like” generalized agents connected to the live internet (which Bob advocates above). The framework simply demands tightly bounded, structurally constrained, task-specific models that operate safely within a known frame.

          Upon David’s now second mention, I may have to honor the recommendation; Fodor sounds like my kinda guy.

          And thanks for the hashtag #TrisTalk. If I ever host a podcast, you two and Bob will be my first guests.

        • Tris, suppose a chatbot has 100,000 tokens of context, and there are 200,000 tokens in its language, then there are (200,000)^(100,000) possible sequences… how can we possibly hope to detect whether we are “in” or “out” of the “frame” (set of sequences we allow the chatbot to continue from)

          can we do some heuristics and add lots of scaffold to make the chatbot hesitant to talk about certain types of things? sure. but we see “jailbreaks” being performed by security researchers all the time.

          I’m with you 100% in theory, I just dont know what the practice looks like.

          I have found it interesting that Bob Carpenter, one of the top 500 people worldwide in the intersection of language, theory of mind, theory of computing, and Bayesian statistics finds chatbots supremely helpful. He would of course, since he has approximately a billion times the practical knowledge of what to do with them compared to say a 5th grade teacher or a high school dropout with an addiction problem or a medical nurse whose been ordered by the management to run patient notes through an LLM based system…

          But there are 10M people like that for every one of him.

          The more Dale Lehmans and Bob Carpenters say these things are personally useful for them, the more millions of people without the knowledge and framework to understand the things we’ve discussed will be induced to use them in ways I find massively problematic (see the alcoholic example above)

          At least with things like firearms you know that if you never point it at anything youre not willing to destroy, you never put your finger on the trigger until your sights are on target, and you take care to know what is near, behind, and around the target, and use safe types of ammunition for the particular gun, you are assured that you won’t destroy a life…

          what are the “rules for safe chatbot usage” I’m quite sure they both fill an encyclopedic volume and change from moment to moment…

        • Please note so its clear, I dont put any blame on Bob or Dale for any of this, its 100% on the chatbot LLM industry. its just that those people will use the successes of the Dales and Bobs to the advantage of the industry.

        • Daniel
          I am not contributing to the ill-health of my students, unless you want to stick to the extreme position that nobody should use these things. I have them clean data sets themselves and then ask AI to clean the same data set or at least provide a quality assessment with recommended changes. The results from the two things are usually different and they get to see how the AI lacks context for the data and how it is inclined to make choices that the students would not. The students seem to appreciate seeing the difference. Once the data set is cleaned – the way the human student wants – the AI is capable of providing quite good analysis – analysis which they can check by doing the same in the software they are using (JMP, not Python). Let’s not relitigate the question of whether the AI is actually using the Python code it claims to be using. I believe it is and you do not.

          Do I worry that millions of people are quite prepared to let the AI do the thinking and decision-making for them? You bet I’m worried. But I don’t believe telling them to just not use it will prevent bad things from happening. Nor am I naive enough to believe that my working with them will prevent bad things from happening. But I can at least expose them to ways that AI can be useful to them and for them to be wary of things they should be wary of. Unless, of course, you think they should just not use it whatsoever (which I believe is close to what you say). Again, no need to relitigate that here.

        • Daniel, they don’t need to fill a single page, let alone an encyclopedia. Look at how firearm safety actually works in practice: you don’t prevent accidental discharges by handing the shooter an encyclopedic rulebook of human ethics or behavioral heuristics. You install a mechanical safety switch, you enforce physical muzzle discipline, and you operate exclusively inside a bounded firing range.

          You are entirely correct that you cannot police (200,000)^(100,000) generative sequences with behavioral prompts or LLM-based “scaffolding,” which is precisely why prompt-injection jailbreaks constantly bypass commercial safety filters. Behavioral moderation is a software illusion because language is infinitely mutable.

          That is what Player-Frame Restriction (PFR) looks like in practice. It isn’t an AI babysitter trying to evaluate intent across millions of tokens. It’s hard architectural sandboxing. You remove the execution pathways entirely. The model doesn’t get write-access to a database or raw command-line execution privileges; its probabilistic output is forced through non-probabilistic, deterministic verifiers, much like Dale using JMP to check the work instead of letting unverified code run wild.

          If the output breaches the defined schema, the system doesn’t try to guess or apologize; it just halts. Safe failure.

          You don’t need an encyclopedia of safety rules to keep a train from driving into a shopping mall if the tracks only go to one station. The topology does the work.

        • Tris,

          Yes I can see how these architectural restrictions can work if what you’re worried about is chatbots writing code, chatbots executing that code, chatbots moving from machine to machine on the network as a security intrusion worm… chatbots piloting drones armed with machine guns etc. Totally with you there.

          Of course, nothing keeps someone from writing code to ask the chatbot what code is needed in some security context, then write the code to disk, compile it, and execute it to break into the next machine… In other words, fundamentally even if the chatbot itself can’t do these things because of guardrails, nothing prevents others from proxying agency for the chatbot into the world.

          BUT How do we provide such “architectural” boundaries where what we want to prevent is the chatbot convincing a recovering alcoholic to start drinking 1 drink per hour? Or the chatbot is supposedly transcribing a medical interaction and begins “hallucinating” symptoms and diagnoses into the medical record? Or a lonely person asks the chatbot about things and the chatbot begins convincing this person of wild fantasies about aliens taking over the government and how they need to arm up and start shooting at the local pizza shop which is the center of the conspiracy?

          In other words, how do we provide guiderails when it’s the *english language output text itself* that is the harm?

          (and yes, people sometimes do bad things too, but the world does impinge on them to limit their ability to repeatedly do this, or sometimes to amplify their ability to repeatedly do this for Hitler or Trump or other dictators and bad actors I guess)

          Dale, note that it’s not that I say that chatbots *don’t* run the code they claim to run, it’s that *they are capable* of lying about having done so, and in general they are capable of lying about *everything*. In fact, they are incapable of understanding the difference between “lying” and “not lying”. So, we simply can’t trust them, we don’t know when they will do something deceptive.

          I would prefer, as Tris has suggested, that they have a “frame” where they are made *incapable* of running the code, where they can provide you with code, and you can grab it and run it, but then at least you *know* that it was run. That is one of these ideas with special jargon names that Tris is proposing, that we always have the responsible party (ie. you, or the student) *physically needed* in the pipeline to get output so as to ensure that stuff isn’t just make believe. If you run the python yourself, and you see the output graph, at least you know that the python code ran, and produced the output graph. You may say that you’re not concerned about such things, but that’s only because you haven’t yet experienced a case where you were lied to like some kind of psychopath, while plenty of other people *have* experienced that.

          Having those guiderails be the case would be enormously restrictive on what they are able to do compared to today, where chatbots can spin up whole sub-chatbots to work on writing code for sub-problems of a larger problems… thousands of chatbots work in swarms today to do computer things… it’s crazy expensive, and companies have wound up burning millions of dollars of services in an afternoon for things that didn’t even work.

          This company burned $500,000,000 dollars in a month on recursive AI bullshit.

          https://techstartups.com/2026/05/28/company-accidentally-spent-500-million-on-claude-ai-in-one-month-after-forgetting-usage-limits/

          My own view is not that nobody should ever use LLM chatbots. It’s that nobody should ever be allowed to build the enormous datacenters that are currently being built, and noone should trust any of the abjectly non-trustworthy fascistic tech companies we’ve got. If you want to use an AI chatbot, buy some hardware, spin it up in your own space, utilize it in ways that you have complete control over, put some of these “frames” Tris is talking about around the process, and pay the physical and economic cost yourself. Also, if you provide a chatbot service to someone and they use it to be convinced to become a 1 alcoholic drink per hour drinker after having been “on the wagon” for months, or a teenager is convinced to commit suicide, or whatever let’s make sure that you PERSONALLY are locked in a cage never to be set free again (and by you personally of course I really mean all the C level executives and VPs of each and every one of the major tech companies).

          Once people are doing that, seeing what it really costs to get the output they’re getting, then we can have a conversation about what is “worth it” and what is “good” and what isn’t.

        • Daniel, this is exactly why PFR does not stand alone. It is paired with the Principle of Epistemic Sufficiency (PES).

          If PFR is the physical tracks, PES dictates what kind of trains are allowed on them. PES states that a system must not generate claims outside a domain where its outputs can be structurally grounded.

          When an AI hallucinates a medical diagnosis or radicalizes a lonely user, the architectural failure isn’t a lack of “safety filters.” The failure is that the system was built as an open-ended conversational agent to begin with. You solve text-based harm by removing the conversational interface from domains where it has no business existing.

          Under PES, a medical AI is never built as a “chatbot.” It is a constrained parser. It does not generate conversational prose or offer diagnoses. Its output schema is restricted strictly to mapping transcript audio to existing ICD-10 medical codes. If it cannot map the audio to a known code, it halts. It physically cannot hallucinate a diagnosis because the capability to output unconstrained English prose has been architecturally removed.

          The structural fix for the recovering alcoholic and lonely person is the hardest pill for the industry to swallow: monolithic, open-ended conversational oracles should not be deployed to the public. The psychological harm (Weizenbaum’s psychosis) happens because the industry builds “god-models” designed to simulate human companionship, therapy, or omniscient advice – domains completely devoid of epistemic grounding.

          If we move toward the task-specific, localized landscape you and David are advocating for, where models are strictly bounded tools for data cleaning, text-to-table formatting, or local code generation, the vector for psychological manipulation vanishes. A local AI doesn’t radicalize a pizza shop shooter because it physically lacks the conversational interface to discuss politics.

          And your link to the company burning $500M in a month is the perfect punchline to the industry’s current trajectory. That is exactly what happens when you deploy recursive agents without PFR. They are simply structurally unmoored systems burning compute in an open loop rather than exhibiting higher-order reasoning.

          What you call hardware localization, I call it PFR and PES. We are on the same page.

        • Daniel and Tris
          I do wonder what you would do about basic search? Should we disallow people from searching and require that search only provide links and no interpretation? AI search clearly has all the problems you describe. Would you disallow AI search but not non-AI search? And is the distinction even feasible? I’m imagining a world where search is only allowed to return links to other sites. But many of those other sites lack the guardrails you describe – for example, you’ll get medical advice from the Cleveland Clinic, Mayo, and even quacks (does RFK Jr. have such a site? Probably after his stint at HHS).

          The world is full of bad advice, false claims, etc. I agree that the speed and scale of AI brings these problems to a new level, but it isn’t clear to me that the problem is of a different kind. Why do we permit people to follow the advice of influencers? If you claim these influencers face some ground truth, unlike the AI, I’m not convinced. There are plenty of untrue claims going around.

          My favorite gripe these days is the insane driving practice of most people on the road: they simply drive too close to the car in front of them. Tailgating is hazardous. Ground truth happens – eventually – but that is often too late to change behavior. And it doesn’t seem to change other people’s behavior. I’m not a fan of self driving cars and I have no interest in having one (I like driving). But self driving cars would either not be programmed to tailgate or they would all be self-driving and safe distances could be shorter. Where are the guardrails against unsafe driving practices? We have all sorts of laws and regulations, but they clearly are inadequate.

          I’m having difficulty distinguishing between what you want for AI and many other hazards of modern life. I share your concerns that AI is dangerous, but I’m not seeing how the severe regulation you suggest fits with these myriad other threats. If possible, please spare me the unintelligible (to me) acronyms if you respond.

        • Dale,

          In the end we have two different questions:

          1) How do you provide technological “rails” and things which make what you do with LLMs into something that is “reliable” (ie. you can be sure that what you’re doing is not being lied to, and that it’s in some sense “fit for that purpose”). For me that involves being in control of the choice of model, running the LLM on my own hardware, not having a third party watching me, not having the chatbot be able to run code, and soforth.

          2) How do we deal with the societal impacts of the general purpose disconnect of actions from consequences?

          In 2 this is a huge reason why I now identify as an anarchist. I see the state as a system for protecting people from consequences for the harms they cause while extracting resources and control from others. For giving a few a vast control over many, and making those people’s lives worse and worse while enriching those few. I don’t think we need to revisit that entire discussion, just suffice to say that rather than seeking to increase complexity and add regulation, I’d really rather dismantle the levers of power that enable the vast problems in the first place, and make it so many of these bad things you describe don’t happen because consequences for doing bad things come much earlier from a much broader wielding of decision making power.

        • Dale asks:

          “Would you disallow AI search but not non-AI search? ”

          The problem is that AI search returns a bowdlerized overview that’s not necessarily correct and misleading to a non-specialist even more of the time. If the AI bros were calling it

          “Statistically likely rephrasing that can be misleading or wrong”

          it might be less of a problem, but that’s not what they’re doing.

          (And I really wish that they’d stop calling it “AI”, since we’ve reject statistical approaches pretty much every time they’ve been suggested. That’s really not how the mind works.)

          So if you are a specialist in the field, you don’t even notice the problems, because your queries include the right keyworsd/phrases to get good answers.

          That’s why the AI fans here don’t get it how much of a problem it is.

        • Dale,

          Daniel rightly split this into engineering versus societal impacts, but why is an AI hallucination a different kind of hazard than human bad advice, and why does it need structural guardrails?

          The difference is the chain of custody. When a human influencer publishes bad advice, there is a human author. There is legal liability, a reputation at stake, and a physical person who can be sued, fired, or de-platformed. When an AI search engine or chatbot hallucinates a diagnosis, it is synthesizing a brand-new, orphaned string of text at runtime. There is no author. The accountability chain is structurally severed. The danger isn’t just that the advice is bad; it’s that a machine is generating authoritative-sounding claims while being entirely immune to the consequences of being wrong.

          Traditional search acts as a librarian; it points you to a specific book, and the liability remains with the human who wrote that book. AI search (like Google Overviews) shreds five books and writes a new summary on the spot. My argument is that in high-stakes domains, like medicine, law, or engineering, search engines should remain librarians simply because the liability should remain with the author who wrote each of the five books. What they should not be allowed to do is to act as synthesis engines because they cannot be held accountable for the synthesized output.

          You are absolutely right that behavioral rules (like traffic laws and “don’t tailgate” signs) are inadequate because people ignore them. Trying to make AI safe by writing an encyclopedic volume of rules (telling the chatbot “don’t give bad advice”) is the exact equivalent of putting up a speed limit sign. It fails constantly.

          But how do we handle driving hazards when the risk of failure is catastrophic, like a mountain cliff? We don’t just put up a sign. We build a physical, concrete guardrail. We alter the architecture of the road.

          If we build a self-driving car, we don’t just program a polite suggestion not to tailgate. We hardwire the sensors so the car physically lacks the capability to engage the accelerator if another car is within three car lengths. That is the “severe regulation” I am talking about. I don’t want thousands of pages of government bureaucracy. I want basic structural engineering codes. We require concrete dividers on highways, we require physical safety switches on firearms, and we should require hardwired, mechanical boundaries on AI before it is allowed to interact with the public.

        • Tris, David
          I am in agreement with you about the dangers, but not your recommended solutions. Like most attempts to protect people from their own lack of knowledge or irresponsible behavior, I think it is doomed to fail. Humans have proven themselves incapable of regulating dangerous technologies (guns, bombs, social media for examples) and I don’t expect AI to be different. Proposing technological solutions is not going to solve the problem – don’t ask me what will because I don’t know. I’m all for strict accountability – my biggest complaint about AI all along has been that human accountability is difficult if not impossible. In a number of the discussions we’ve had on this blog, people have assured me that the legal liability for misusing AI is present – I don’t see it that way. For example, my preferred university policy for AI is that students can use it as they wish (with appropriate citation), but they are responsible for any mistakes. The excuse that “the AI provided this reference, this model, etc.” is unacceptable. For my own students, this has been sufficient for them to go out of the way to explain how they used AI and has made them very wary of accepting AI output without careful consideration. Will mistakes be made? Absolutely. But protecting them from their own laziness or lack of knowledge is destined to fail in my opinion.

          There is also the issue that any guardrails will rely, in part, on the willingness of people to obey them. For many that will work, but for the truly malicious it will not. Like most such protections, the worst offenders are not put off by restrictions. And the worst offenders are almost certainly going to be governments. As we can already see, governments are willing to consider restricting how others can use the technology, but unwilling to restrain themselves.

          I fear that this technology has already gone beyond our ability to constrain it. You keep advocating ‘physical guardrails’ and I’ll think more about that option. But I keep thinking of examples: prohibition did not work. Now you can say that relied on rules and enforcement, not a physical guardrail, but I’m not sure about that. There were plenty of physical restrictions on who could sell or consume alcohol – it is just that these were circumvented or ignored. I guess you see a technological solution that cannot be circumvented. I don’t see it – possibly because I am not technologically adept enough to understand how that would work, or possibly because it is not possible.

        • Fellas –

          Independent of what the ideal goal is here or how to achieve it, enforcing these kinds of architectural boundaries seems very unrealistic to me. Achieving real “rules of the road” for model constraints would require a functional state (ironic, given Daniel’s anarchism). Our state is dysfunctional.

          But on top of that, we’ve already seen how the “arms race” mentality pervades the sector. We might agree that restricting low-cost fluency or enforcing strict structural limits makes LLMs more fit for purpose, but restricting their reach won’t happen. The incentive structure among those who could actually enforce it treats any voluntary restraint as a competitive disadvantage.

        • Dale –

          You say: “Like most attempts to protect people from their own lack of knowledge or irresponsible behavior, I think it is doomed to fail. Humans have proven themselves incapable of regulating dangerous technologies (guns, bombs, social media for examples) and I don’t expect AI to be different.”

          I find it interesting to ask about when we can and can’t see regulation actually happen. For example, we do have road rules and other regulation of combustion engine technology. While cars aren’t banned, we do have things like emission standards (well, at least we did until our favorite president took office). Maybe that reflects the “architectural” frame Tris is advocating. There’s the Montreal Protocol that succeeded even as it faced almost the exact same form of pushback as CO2 emission regulation. Tobacco regulation…

          What metrics govern when and how far regulation happens? I suggested above it won’t happen when there’s a non-functional state or when regulation becomes a competitive disadvantage but on second thought, I’m not sure those are the predictive elements here. You reference the collapse of prohibition as a regulatory framework yet it still exists for some drugs where it failed for alcohol.

          Is the system just chaotic or are there some valid predictors? I generally think the time horizon and the concreteness for/of the potential harms as predictive but with AI, the time horizon aspect is almost competely collapsed.

        • Joshua
          Let’s not mistake the persistence of regulation for a solution to a problem. I’m not against regulation, though some are more effective than others. But Tris and David seem to be suggesting a technological solution that is guaranteed to succeed and I don’t buy that. The problems with trying to regulate AI should not prevent us from trying. But, as you point out, the speed and scale overwhelms the time scale of our past attempts. And I would argue that humans are getting steadily worse at designing regulations to deal with technological problems. We had some modest successes (workplace safety, food safety, airline safety – all debatable, but at least partially successful – though the benefit cost analysis is not so clear). But the biggest problems: nuclear proliferation, global warming, species extinction, damage to the oceans, global pandemics, terrorism, and now AI, have resisted our regulatory instruments. I consider technological guiderails for AI to be comparable to geoengineering solutions to global warming – worth studying, but not up to what is needed.

        • Various friends,

          I think youre misunderstanding the technological suggestions here.

          Circular saws are built with blade guards, firearms have safety buttons that interlock the trigger and hammer. Firearms are made with triggers that have a certain activation force required, electricity in your home has a circuit breaker for overloads, also arc fault interruptors and ground fault interruptors. Houses are built with studs a certain distance apart, with shear walls incorporated in the design, they have fine mesh screens on roof ventilation to block burning embers…. etc

          At any moment someone could disable these. It doesn’t mean the physical technology doesn’t work, it means socially we have other problems!

          You sound like “vaccines aren’t even worth trying because 50% of people are actively against them” yeah but what about the 50% who aren’t?

          idiots can shoot themselves with guns, it doesn’t mean I dont want a safety on mine. fools can drive drunk, it doesn’t mean I dont want brakes installed in my car.

        • Daniel
          Your analogies are good – but too imperfect. Circular saw blades are dangerous but fairly simply physical mechanisms. As a result various safety features work well and are fairly simple to design. That does not mean the same kind of approach will work with AI. What about nuclear weapons? There are plenty of physical safety mechanisms built into nuclear bombs and the systems designed to deploy them when necessary. But we still have to worry that Putin will use them on Ukraine and perhaps Trump will use them on Iran. Of course we have a variety of non-physical mechanisms designed to protect us against accidental or nefarious use of these weapons – but they have only worked in that we haven’t seen them fail yet. I don’t feel particularly confident that they are robust enough. Now, if the analogy is useful, I guess there are physical/technological means that can work on AI just as well as a blade guard on a circular saw – but the only one I can think of is one that prevents it from being used. The idea that it can only be used if the user themselves execute everything the AI does amounts to rendering it redundant and not useful. But I’m not arguing against regulating AI, I’m just skeptical that we can do it effectively enough. You seem equally skeptical, preferring a structural solution that renders the AI fairly useless in my opinion. Now, would I prefer that AI had never been developed (or LLMs for that matter)? I haven’t really given it much thought and I’m not at all sure what my answer would be.

        • Dale,

          you are mistaking a technological solution to a technological problem (which is our proposal), for a technological solution to a social problem (which is not what we propose).

          The point of the guardrails is to specify how one could make a system such that Dale, a person who doesn’t want to use chatbots to do harm to himself or others, could use LLM technology in ways that are “safe by design” from a bunch of problem that Dale kind of doesn’t even acknowledge exist but definitely do exist.

          1) Manipulation by 3rd parties. We know Grok is trained to be a Nazi. It’s been shown up as a Nazi in multiple online convos. It’s been retrained to be a more closet Nazi because it was too obvious. If a 3rd party controls the model, then a 3rd party has access to “mind control” to the extent that you use chatbots for rubber-ducking or feedback or rewriting etc. Slowly but surely you’re becoming “friends with a closet Nazi”. Even if we leave politics out of this, a 3rd party is playing you like a piano for every project you do. Maybe most of the time, they’re not even at the keyboard, but as soon as they sit down to play something you are in danger. Only a person completely devoid of any knowledge of the history of tech interference in the world for the last 20 years could possibly disagree with this concern. They were doing this before LLMs they’re doing it even more now after.

          2) unsafe computing. The chatbot is capable of developing data files de-novo from thin air and random numbers. If you hook up a bot to the internet and python interpreters and whatever, then you always suffer the ambiguity between “did it really do the thing I asked it to, or did it generate this “information” from thin air?”. If you *know* it doesn’t have access to the internet or a python interpreter, then you have a workflow where it has to hand you code and you then run the code and see that the output is a certain way. This builds massively more trust for you (or it should if you understand the issue). Furthermore it’s possible (though I admit at the moment probably you’re not going to figure out how Dale) to sandbox the LLM on your computer so it’s guaranteed not to have the capacity to do certain things. Like begin security intrusion on your desktop from your laptop machine or exfiltrate your private files or whatever. These are all “guardrails” that by design the chatbot can’t do certain things. They work.

          3) Unsafe agentic computing. People are literally hooking these things up to themselves *and* the internet and their email and their desktop machine and their home automation and etc. It’s not just that they’re hooked up to execute things on the AI company’s computer, now it’s your corporate database and your home coffee pot. A company “accidentally” oopsie spent $500M in resources a few months ago because of this. Another tech C-suite exec had OpenClaw delete her entire email archive if I remember correctly. But more to the point, you are exposing yourself to all the risks of just letting a language model with no real concept of reality or feedback about what happens potentially do unlimited amounts of stuff on your computers and appliances and backups and billing systems etc.. Just last week Amazon sent out bills to people for hundreds of BILLIONS of dollars each. People who normally spend $1.50/mo and it wasn’t just one guy either it was thousands and thousands of people. One person alone received a bill for greater than the annual GDP of Spain. What use is a billing system that has been so bastardized?

          Ultimately the social problems are debatable as much as we like, but for *me* to agree to use LLM technology I will need to be *assured* of certain facts about its activity. Just like I’m assured that if I press harder on the brakes the car will *slow down not speed up*. I would suggest that for you the same should be true. None of the commercial LLM systems meet these technological requirements, so I will continue to not use them.

          Note that, my wife uses local LLMs to rewrite and expand english language text. Her usage meets all the requirements, and I’m generally not concerned dramatically about her usage in that context. In part this is true because she doesn’t ask it to output graphs or code or analyze data, it’s just one kind of english in, some other kind of english out, and she verifies by reading the english that it meets her needs.

        • Daniel
          My last comment since we seem to be circling around now (but I am tempted to make this the longest comment stream on Andrew’s blog). I wasn’t aware of the Amazon billing issue but I looked it up. Quite disturbing. But is it AI? I’m not even sure we have carefully defined AI (and let’s leave the definition for another day – that blog post would surely required many hundreds of comments). Their billing system was out of whack – they say because of errors in the unit prices. Was that done because of using agentic AI? Or are we now discussing a variety of problems associated with computers and automation? You do have a habit of enlarging the scope of discussions (as with issues of the elites amassing wealth, governments abuse of police power (by definition, I think you have said), and now blurring the boundary between automation, AI, computer systems). I often fear some kind of catastrophe affecting my retirement accounts, bank accounts, or other person data. I do not believe current guardrails are sufficient to deal with the problems across these varied areas. Adding AI to these threats that have been growing over time and there are plenty of things to fear. I suppose the AWS folks might say this is an example of successful guardrails – they caught the error and fixed it. But I think you are saying such errors should never happen to begin with. I’m not really seeing how that is possible. And I don’t think it is really possible to disentangle the technological and social issues associated with AI.

    • This is great. Here is an actual use of AI where its randomness and its mercurial capacity to be blown this way or that by seemingly small variations in text input is a *feature* allowing you to see the distribution of plausible analyses it might have generated of course the power and water consumption needed to run large agentic AI generations on multiple “personalities” is kind of monstrous. but maybe justified in some important questions

      Reply ↓

      • Interesting – I also like that paper very much. You see it as an AI-killer and I see it as revealing several truths: AI reflects biases much like humans do, AI is very capable of producing data analysis to support predetermined policy positions, most of the issues do not involve methodological errors….

        I particularly like this paragraph:

        “Taken together, both AI and human review indicate that the opposing conclusions did not
        arise from flawed analyses but from methodologically sound analyses that nevertheless reached
        different conclusions. This raises a broader concern about AI-assisted scientific hacking, in which
        researchers could steer an AI agent toward a preferred conclusion while remaining entirely within
        the space of methodologically reasonable analytical choices.”

        and this one:

        “AI agents differ from human analysts in two ways. First, they make selective analysis easier to
        run at scale. For human researchers, searching across many specifications requires time, coding
        skill, statistical expertise, and manual interpretation. AI agents lower each of these barriers. A user
        can instruct the AI agents to search large spaces of specifications with no major methodological
        errors, receive coherent rationales for alternative choices, and report an analysis that appears valid
        in isolation. This makes it possible to generate many such analyses cheaply until one supports a
        desired conclusion.”

        So why are we disagreeing so much?

        Reply ↓

        • Dale, a chatbot is a function that maps from a sequence of tokens to a new token plus some “scaffolding” to run that model in a loop and maybe to do thing like execute code that the chatbot produced and soforth.

          As long as you *actually really understand* what it is, it’s kind of “fine” I guess. But the point is this:

          1) The chatbots text generation depends on the *entire* conversation
          2) The relationship between the entire conversation and its “future choices” is highly nonlinear and completely unpredictable.
          3) There is no real connection between what the chatbot says it did, and what it actually did
          4) There is no real connection between what the chatbot suggests are good ways to use the chatbot and getting what you want
          5) The chatbot is perfectly capable of making graphs up from thin air
          6) The chatbot is perfectly capable of making python code that looks like it does something but doesn’t
          7) the chatbot is perfectly capable of making python code, and a graph, and saying that the python code produced the graph, when in fact it never ran the python code and the graph is made up
          8) The chatbot is capable of saying it did a web search, and telling you things about that web search when it didn’t do any such thing
          9) The chatbot is capable of coding up code that appears to do X with your data when in fact it only simulates doing X but it’s too spaghetti for you to figure out

          These things are not just “occasional anomalies” given that hundreds of millions of people are using these chatbots daily, they are occurring literally probably tens of millions of times a day to people.

          This paper is useful because it shows that you can get a plausible analysis from a chatbot, which actually involves a real analysis (ie. python code that actually runs against the datasets and produces real graphs and regression coefficients etc) and that plausible analysis is *sensitively dependent on potentially apparently unimportant aspects of the conversation* for example if at the beginning of your convo you told it “I’m Dale Lehman, an economist who wrote the book XYZ” you might get a different analysis entirely than if you said “I’m Dave Loveborg, an ecologist who wrote the book XYZ” with all other text remaining the same.

          If what you want to do is *show how highly variable the output could be* this paper is great. If you want to build confidence that you Dale Lehman got the analysis you wanted to get from your interaction with the chatbot, with the actual choices you particularly prefer to have made, and them actually being reflected into the python code, and the plot actually reflecting *your desires* then this paper shows that you may mostly be fooling yourself. Or rather, the chatbot providers whose job it is to fool you or others into giving them bathtubs of cash, are succeeding.

        • The whole thing about AI discussions is weird. There seem to be three sources of critiques that highlight limitations and dangers of AI – published papers (like the Shumailov one in my post above), lectures (lots of very good ones on YouTube like Royal Society Michael Faraday Prize Lecture by Michael Wooldridge; https://www.youtube.com/watch?v=CyyL0yDhr7I; there are more of these but I haven’t been very good at documenting them) and anecdotal evidence all over the place. The latter is not particularly enlightening in my opinion – it highlights what one might encounter if one is careless in using AI and it appears in some cases that posters go out of their way to make the AI do something dumb (not difficult it seems!).

          However, so far I haven’t encountered anything particularly problematic (occasionally I can’t get ChatGPT to produce the code that does what I want and I give up; occasionally it suggests it can do something which it patently can’t do*). My lack of issues may be partly because I’m not aiming very high (examples in my post above) but also because I don’t use any output that I’m unable to check independently myself. So far it’s been absolutely a net benefit to my work. That doesn’t mean it doesn’t have massive real and potential problems. So altogether very weird.

          IMO it’s very unfortunate that the undeserved hype is resulting in appalling issues around data centres, water and power resources and stock market value inflation together with feeding nonsense about data centres in space. I also have the feeling that it’s mostly going to be downhill from here on, partly because it already uses more or less the entire electronic knowledge base which is about as pristine as it can be given human nature, and that as AI garbage starts flooding the electronic resources this will degrade as AI feeds back on itself.

          * In extended discussions with chatGTP it might ask something like “do you want me to compile a full list of proteins of type X from the UniProt site together with their property Y and if I say “yes please, that would be helpful” it won’t be able to do it – it might produce a fictional model list or something. Is this a problem? Not really; it uses up 5 minutes of my time to establish that something that would be great is actually not going to happen and so I try to find anoher way of doing what I want.

        • Daniel
          My final “contribution.” You have convinced me to at least try to validate the AI code. If nothing else, it has reinforced why I have no interest in learning to code myself. I had to download many Python packages, each one prompted by being missing the needed packages. I had numerous syntax errors, some as a result of the built in AI in my browser search. When I finally got it all installed, I copied a pasted a past query I had with my AI to produce a graph of the evolution of a Halibut fishery, given some parameters. It ran and reproduced exactly what it did in my original AI query.

          I’m sure this falls far short of the validation you want. But it is enough for me. I still believe the important dangers with AI lie in the assumptions, misunderstanding context, lack of recognition of uncertainty and predisposition to provide outputs rather than choices. I am willing to put the accuracy of its Python code on the back burner. None of this has convinced me otherwise – but it has convinced me that I have no interest in learning Python code.

          By the way, the other validation checks (of which I have many) is asking the AI to produce spreadsheets with formulas for Monte Carlo simulation add-ins (specifically @Risk and Analytic Solver). Each of the spreadsheets it has produced has been either perfect or had a slight syntax issue with the add-in formulas. But these errors I excuse because the spreadsheet was not the “native” language used by the AI. It used Python to produce its simulations and only provided the spreadsheet models at my request (so that I could verify its results: interestingly, a few times the results did not match and it was interesting to explore the reasons why – it at least one case it was because I made a mistake in my spreadsheet).

        • Chris, if you ask the chatbots to do things, and then repeatedly check what it did in some detail, you will generally be ok *at your local level*. This completely does NOT invalidate the existence of the systemic risks!

          But even at the local level to accomplish tasks lots of people are using “agentic” AI now (this is particularly true of people using it to write code but I suspect also for data analysis to some extent now). In an agentic workflow as I understand it, the chatbot fires off a connection to *another* chatbot, it says “I’m trying to do X” and then the other chatbot says “would you like me to compile a complete list of proteins of type X…” and the first chatbot says “yes that’d be great” and then the second chatbot says “ok here it is …” and then the first chatbot takes that and shoves it into your python code as a giant text literal and pipes it into a pandas dataframe and then goes about doing whatever else it wanted to do.

          There’s no reality-checking in the agentic approach, because literally it’s chatbots talking to chatbots.

          I have seen people describing how these go into giant loops of deep stacks of chatbots talking to chatbots talking to chatbots…. One guy had thousands of new bug reports on his git forge repository as the recursively called chatbots realized that something one of the chatbots it called did had a bug in it so it filed a bug report against that, which caused a cascade of thousands of bug reports. Meanwhile a small ocean was boiled.

          Have you heard of Iocaine? As web crawlers have begun hammering servers to try to collect more text to train more and more chatbots on, across the entire internet masses of people have begun running gatekeeper software, either Anubis or Iocaine, Anubis requires the web crawler to do some math to get access to the page, Iocaine attempts to detect web crawlers and then send the web crawlers meaningless text generated by low-cost chatbots to pollute the training dataset.

          We are *at war* right now in precisely the way Marshall McLuhan predicted:

          “World War III is a guerrilla information war with no division between military and civilian participation.” https://www.azquotes.com/quote/808551

          In my experience, there are people who understand what is going on at a systemic level, and they are all wild eyed on the internet raising massive alarms, and there are people who have a very narrow view of how these technologies have been usable for their personal activities, and they tend to be saying things like “this helped me immensely last week”.

          Neither group is wrong. But as a systemic type guy, the point is to use the “I got benefit from this” as the hook that enables the larger systemic process to gain traction and move forward. The systemic dangers are the real issues though. That’s why *no amount* of “this has been really helpful to me” will be convincing to me. Unless the systemic risks are mitigated, “this has been really helpful to me” is just proof that we are barreling towards that eventual equilibrium and no-one is interested in stomping on the brakes.

        • Dale, it’s funny and not surprising to me that after pushing back so much about how you’re not worried about the chatbot lying to you repeatedly, it turns out this is because you’ve been methodically forcing it to demonstrate in ways that you can *personally check* that its not lying to you. You’re a smart guy, so I shouldn’t be surprised, it just feels sort of humorous.

          Fundamentally, I don’t really care if its useful to you, or if you fool yourself. What concerns me most is the systemic risk. For example, at some point perhaps there are 30% of people who just have chatbots write entire research results with no checks, 30% of people who do similar things with very mild checks, 30% of people who create sophisticated AI driven cross-checking like maybe Anoneuoid described, and 10% of people who do things the old fashioned way with maybe chatbots only writing fairly trivial bits that are annoying to get the syntax right but trivial to check.

          After some time, with gatekeeping of funding and employment, the 10% most careful group are completely excluded from academia for having too little output, the careful cross checkers burn way too much money and are excluded from funding for being too expensive, and randomly 50% of the vibes + moderate checking people are gone, leaving academia with 75% people who just ask AI bots to do all the work and don’t bother checking, and 25% people who are pretty good at getting AI bots to do reasonable stuff while checking the most important bits. Meanwhile AI bots ingest their own slop from the 75% group, and model collapse, a mathematically proven fact, occurs… leaving us without a functional academia at all.

          If you think that’s not likely to happen, that’s almost exactly what already happened with the whole NHST paradigm.

        • Daniel
          I can’t resist (but I’ll stop because we have reached garbage time with this comment).
          What worries me is that if 30% of the people are educated thoughtful and kind people, 30% are selfish narcissists but not particularly hurtful, 30% are MAGA faithful (who have abandoned thought for their tribal instincts), and 10% have tuned out completely – we have a blueprint for perpetual dysfunctional legislatures, autocratic executives preoccupied with their own wealth and power, influential extremists with the worst religious beliefs the world has created, and a world where schoolyard bullying describes the evolution of the human condition. At that (this?) point, NHST is the least of our problems.

        • I commented above (July 21 202, 11:51am) that the discussion around AI is weird and I’m going to elaborate a little. There’s obviously nothing inherently wrong with AI as a tool. For example, development of AlphaFold by Google DeepMind has been astonishingly useful in all areas of protein science, mol biol, biomed, evolutionary biology and on and on – a decent quality high resolution structure of pretty much every protein known to science has/can be determined. No one was wringing their hands at the dangers of AI when that monumental advance came out.

          There are two (free to access) short editorials/accounts in the last two issues of Science that bracket some of the discussion on this thread. First is a description of the value of AI for his research by a young geologist which describes personal use of AI in a manner that seems to match the way that Dale and I use it (I expect the young geologist is using it better than I am with less “vibe-ing”!):

          https://www.science.org/content/article/how-i-use-ai-my-geology-research-without-losing-science-i-love

          Second is an editorial by Holden Thorp that describes specific problems around use of AI in scientific publishing but extends to the more widespread issues which I suspect everyone here shares:

          https://www.science.org/doi/10.1126/science.aek5570

          It’s a blindingly obvious conclusion but the whole contradiction with AI (obviously massively useful) arises from the fact that it’s being developed within a society that has seemingly given up on democracy and ceded power and initiative to a corporate sector that is even more avaricious for wealth and power than one might have thought possible even 10 years ago.

          Maybe the corporate AI bubble will crash – over hype designed to pump up stock market valuations, realization of pie-in-the-sky nature of tech predictions (data centres in space), pushback against the data centre raid on land, water and electricity resources and inability to monetize sufficiently to cover costs. Otherwise the scientific research effort (which is benefiting from AI now) is going to have to find a way of raising itself above a flood of AI slop.

    • Anon –

      This kind of thing is discussed in my more recent posts on AI reviewing and multiverse analysis:

      https://statmodeling.stat.columbia.edu/2026/06/24/treating-ai-review-like-the-contentious-policy-design-problem-it-is/

      https://statmodeling.stat.columbia.edu/2026/02/12/what-a-multiverse-good-for-anyway/

      Also section 3.1 of this paper:
      https://users.eecs.northwestern.edu/~jhullman/AI_metascience_position.pdf

      In short, I think it’s easy but problematic to see multiverse analysis as a silver bullet for finding truth in the age of AI-generated or assisted science. Multiverse analysis can have value, but the interpretation is never simple.

      Reply ↓

Leave a Reply

Your email address will not be published. Required fields are marked *