This is Jessica. I spent last week at a workshop where we were asked to prepare a short provocation related to interactive data analysis (same workshop Andrew mentioned already). In thinking about what could be said about the future of data analysis on the way there, I decided one can’t really consider the future of data analysis, including how to address issues of forking paths and replicability, without considering LLMs.
This seemed like the right direction given that they asked for a provocation. After all, what better way to put people on edge these days than the cliche and annoying move of changing the topic of what is meant to be a serious academic conversation to focus on LLMs?
But I was also being sincere. After I started thinking about it, the thought of spending a week thinking and talking about the future of interactive data analysis without engaging at all with language models seemed irresponsible in light of the current moment.
For one, LLMs are already being used for scientific purposes beyond just writing and coding suggestions. Summarization and qualitative analysis of text corpora, simulating outcomes of large social systems, generating digital twins for individuals to estimate individual-level counterfactuals in medical trials, and so on. And so naturally researchers are exploring their use as general purpose tools for data-driven analysis and visualization. Wrapper systems that take a high level analysis goal like “analyze last year’s sales data” and interface with an LLM (and access to other statistical tools, e.g., for fitting the models) for the user are being developed.
And some people are already doing exploratory forms of data analysis with them. For instance, OpenAI has a page on their site dedicated to their expanding features for ChatGPT-supported data analysis, with quotes like this:
ChatGPT is part of my toolkit for analyzing customer data, which has become too large and complex for Excel. It helps me sift through massive datasets, allowing me to conduct more data exploration on my own and reduce the time it takes to reach valuable insights.
Or this:
ChatGPT walks me through data analysis and helps me better understand insights. It makes my job more fulfilling, helps me learn, and frees up my time to focus on more strategic parts of my job.
To many people who care a lot about data analysis this may seem horrifying. But increasing the level of automation in data analysis workflows has been happening for years. It’s often our goal in our own personal practice to automate the tedious steps of modeling workflow and identify patterns and protocols to guide us through the harder decisions. So I find it interesting to reflect on why we might react especially negatively to the idea of incorporating greater levels of assistance and automation via LLMs.
As a thinking prompt, imagine yourself before you knew much about stats. You have some data to analyze and can get help from either:
- A rule-driven assistant that has you answer some multiple choice questions about your goals and data spec and then recommends a modeling approach or test
- A (human) statistics consultant with an advanced stats degree who has a conversation with you then makes recommendations and is available to provide review as you complete your analysis.
- An LLM that attempts to perform the same role as the human consultant.
What makes these options seem different, and do any of these differences lead us to believe one is much better than the others?
One difference would seem to be that two of these things are blackboxes, which we might expect to be more unpredictable in output than the other. It seems easier to evaluate how #1 could fail. Its specification is more concise and interpretable. We would expect #2 and #3 to produce output for a much wider range of inputs, and the outputs might vary a bit depending on how you approach them, when you ask them, etc., making evaluation harder. Comparing the human consultant to the LLM, we would expect the LLM to have ingested a lot more examples, good and bad, and a lot more statistics texts. There’s potentially a huge amount of signal and a huge amount of noise in all this, and we don’t understand fully how it’s getting combined. The human’s training data and synthesis process is also hard to delineate. But maybe still more debuggable in that we might expect them to respond based on some smaller set of principles or protocols or heuristics that they refer to to organize their behavior.
There are many other dimensions we could compare them along – learning rate, awareness of conventions across different domains, etc. Are any of the dimensions on which they vary enough to make one of them seem necessarily better? Do we believe that whatever level of machine assistance we are using now is the optimal level, and that an LLM couldn’t improve upon it? I don’t necessarily have answers or even strong opinions, but I find it interesting to think about.
I suspect some of the resistance to LLMs as analysis assistants stems from fears of further ratcheting up mindlessness in data analysis. We know that regardless of our noble attempts as researchers to educate on what good modeling practice looks like, many people seem to prefer a ritualized approach where statistical analysis is a black box that takes in data and spits out answers. LLMs are threatening because they would seem to make this kind of approach easier. For example, dealing with data formatting issues remains a major time-sink in analysis, especially for users who are not programmers, but ChatGPT at least superficially seems tolerant of whatever you want to paste in. This isn’t to say it won’t err to different extents with different formats, prompts etc., just that the interface is more opaque, which can give a sense of troubles being alleviated.
There’s also fear of “lowest common denominator” analyses, where the user pastes in their data and question and the model recommends or runs whatever is most popular. And of “poisoning the well” when people publish AI-assisted analysis results that exemplify this “data analysis for the masses” vibe which get pumped back into the model as training data.
Compared to code suggestions, which is where they are most likely to be used in modeling workflows today, looking to them for data analysis seems closer to letting them tell us what to believe (since learning from data is often about updating our beliefs).
I wonder though how much the hesitance to consider AI-driven data analysis is the knee-jerk reaction that somehow human domain knowledge needs to be unrestricted and its application left open-ended, that anything iterative and interactive must be left to human control. As if even if we can better attain our stated scientific goals with AI, it will ultimately somehow still be worse. I sense this attitude in fields like human-computer interaction and data visualization, where people sometimes resist the thought of formalizing the goals of some human-computer interaction, as if that automatically equates to taking agency from the human. When it comes to topics like visual data analysis it’s like we would prefer to just trust that the human knows what they are doing and will apply their domain knowledge in the most appropriate ways.
This ignores that when we leave things fully to the human at the highest level, that is also a design choice, one that usually makes problems harder to find because we’re less clear on what we’re looking for. So I tend to think that when the thought of formalizing or automating some task makes us prickly, it’s probably something we could learn from thinking more about. If nothing else, the exercise of trying to imagine a data analysis assistant that optimally pairs human domain knowledge with AI-driven suggestions forces us to reflect a little harder on what exactly we think the human is doing/adding at each step, which is why I find it generative for thinking about the future of data analysis tools.
I recall Andrew saying something during the workshop along the lines of ‘When it comes to data analysis we all like to think that our particular combination of flexibility and rigor is somehow the right balance.’ Which implies that we don’t think it’s obvious how to combine domain knowledge with methods, and we don’t necessarily trust others to figure it out. Why would we want to add another agent we don’t fully understand into the mix? Perhaps it’s natural to hesitate a bit when we sense boundaries in our knowledge, because we don’t want to make mistakes.
Ultimately, I’m not advocating for everyone working on data analysis tools to start thinking about LLMs. But I also don’t think we should let our knee-jerk resistance keep us from considering the bigger picture. Presumably there are also many ways in which they could help human analysts overcome limitations. For example, lately I’ve been thinking about how a paradox of learning from data is that you can’t do it without a good imagination. Many of the human errors we point out in data analysis can be attributed to a lack of ability to entertain multiple possibilities. We like to suppress and reduce uncertainty, not maintain it as we go. We don’t do sensitivity analyses as much as we should, nor take the results as seriously as we should. We don’t engage deeply with many of the assumptions behind standard choices. We regularly ignore forms of multiplicity, from the fact that the same ATE can be consistent with very different patterns of heterogeneity at the individual level to the way a machine learning pipeline can return a set of seemingly equivalently performing models.
Part of the problem is that analysis is cognitive demanding, and there’s only so much we can keep in mind at once. Remaining aware of how different assumptions made along the way subtly impact the interpretation of results might be too high a bar for a person. Maybe designing LLM-based assistants with the express goal of helping us keep track of and critique assumptions, uncertainties, multiplicity, etc., and interrupt us with them at the right times could help curb overinterpretation of results.
It could also be interesting to think about how multiple agents who perform different roles (and get different access to different parts of the problem, including the data) could be combined to get around the various problems of leakage or data conditioning that we see in practice. The challenge is not boxing ourselves in by assuming overly strict designs, but that’s been true for all attempts to integrate more automation in data analysis.
Of course, simulating nuanced use of imagination to drive analysis will require learning what it looks like from somewhere. Ultimately factors like the availability of good prior analyses and our lack of sufficient control in how models are fine-tuned may be major challenges. I’m curious what else.
P.S. It occurs to me this post is partly about how different people react when they see something in their domain of expertise that could be a train wreck starting to occur. Do you jump in and try to redirect it, or avoid the situation altogether? I guess in this case I’m advocating that at least some of the experts jump in.
Much as I think LLMs rival human abilities for many things, I admit that I am reluctant to use them for my own data analysis. I’ve resisted simpler automated tools in the past – for data visualization, I abhor the software “recommendations” that many software products provide for such things. I find that every data set is unique and I also know my own knowledge is severely limited. So, it is a matter of deciding the best way to proceed: invest energy in exploring the data more fully, learning more about methods that I don’t know well, asking experts for help, reading related publications to gain new insights, or seeking assistance from an LLM. Among the choices, I think the returns from using the LLM are the lowest of these. That may not be true for everyone. If I knew less (is that possible?), maybe the cost/benefit calculation would be different. If I did coding, then perhaps I would find LLMs a time saver. But for the type of analysis I do, and the type of teaching I do, I just don’t see the LLMs as being a good use of my time.
Perhaps I will need to concede to thinking as a number of others on this blog have repeatedly maintained. I know the LLMs cannot “think” so I don’t see how they can help with unraveling the stories in the data. If I have a specific task to do (e.g., how do I calculate a Brier score?), I can and do use automated tools. But the more meaningful questions: what is actually being measured and how, is the data accurate, is there some transformation that will fit the data in an interesting way, what is the best recoding for a complex nominal variable, etc. I just don’t trust the LLM’s training data to be rich enough to make it worth my time. I don’t think it has been trained to do these things – it has been trained to find patterns from its training data for what I am asking, and I thing the likelihood of getting much of value is not worth the opportunity cost.
I can relate… I haven’t really had the patience to figure out how to make them helpful for most of the tasks I do routinely. But I find data analysis with LLMs an interesting thought experiment nonetheless.
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4760077
some thoughts on LLMs…..
Well, rather than just guessing, maybe you could try it? And use ChatGPT 4, the paid one. It’s cheap. Just remember to be precise in your questions and remember it’s way better at concepts than it is at data lookup.
I find ChatGPT indispensable for just about everything. But then I’m old enough to remember people saying that calculators would rot kids’ brains, the internet is the devil’s work, and Wikipedia is garbage. This too, shall pass.
Oh I’ve definitely tried it, and was paying for it until recently. I’d probably would use it a lot more if I had more time to program. I just haven’t found myself seeking it out repeatedly for much. So far I’ve found it good for little things but not crucial for anything really critical.
.
Given that some people (like yourself) are so enthusiastic, I conclude that I must not be trying hard enough. Admittedly I am not a super patient person, and I like to generate a lot of things from scratch.
Btw, it would be cool if you could write a post Bob about how it’s helped you critique assumptions or otherwise keep in mind uncertainty in tasks that require deep thinking. Or if you already have written such posts, can you point me to them?
Hi, Jessica. I was replying to Dale Lehman. This is a reply to you.
What do you mean by “from scratch?” Do you use pre-packaged tools like ggplot, or are you like Andrew and want to do everything in base graphics because you have more control? Do you let the lm() function do linear regressions in R or do you work out the linear algebra and code it up from scratch? Do you look at what other people have done on StackOverflow or do you just stick to the language manuals and do everything from scratch?
A bunch of postdocs at Flatiron told me they wouldn’t use Stan or automatic differentiation because it’s a black box and they only trust things they code from scratch. Yet I don’t think they code their own linear algebra libraries.
In the late 1970s, when I was learning to code, you pretty much only had the language manuals and coding took forever. You could ask colleagues how to do things, but the internet wasn’t widely available. In grad school in 1984, I got the internet and then you could post queries and maybe someone would help you out.
When the web came out in the early 1990s, there was widespread mistrust, especially around sites like Wikipedia. People would routinely say they don’t trust the internet and prefer printed materials from libraries. I know people who still use printed maps because they don’t trust Google Maps.
After Wikipedia, which was just a handy reference library, we got things like StackOverflow. It’s really hard to think back to what things were like before that in terms of how interfaces work, how to install things, etc.
I’m a big believer in using all the tools at my disposal to make my life easier. I don’t stop thinking about what I’m doing. If a colleague does something, I don’t just accept it and put it in a paper, I double check it to the best of my ability (I know, I know—that’s not how to maximize one’s bibliometrics—I should have a group of 10 or 15 people cranking out papers at a rate where I can’t keep up reading them—I quit my tenured professor gig at Carnegie Mellon in 1996 when I had about 10 students and felt I couldn’t keep up or do anything myself).
People’s response to calculators, Wikipedia, StackOverflow, and now GPT all sounds very John Henry to me.
Also, I wouldn’t say ChatGPT is a deep thinker on its own. You have to guide it like an assistant. Like humans, it’s much better at processing information than remembering a bunch of facts, as I’ve shown in other responses. I don’t know how to find them or I’d link. I should make a post to summarize.
> I conclude that I must not be trying hard enough
For myself I’m thinking it may be more of a norms thing. Like, what assumptions am I silently making about what is or isn’t a valid scientific/development process.
A few days ago me and ChatGPT wrote some scripts for some very sketchy simulations. It was a weird mix of bash and python and Java and me manually copy-pasting results around everywhere — very not-reproducible, but it got me my results and now I’m thinking about a next iteration (which I can clean up, now that I know the basic experiments are worth doing).
It’s pretty unusual for me to do this sort of thing. I’m pretty clean with code these days (everything in packages, gits, with tests, blah blah). This made me think I’m too pedantic, or at least too focused on writing code in what I’ve convinced myself is the Correct Way. I suspect if I took these leaps of faith into sketchy code more often, I’d be better off. And as far as I can tell, ChatGPT is really good at writing little bits of sketchy code.
I saw a Google spreadsheet someone set up for coordinating some work stuff a few months back. I asked the author to walk me through it — they said they never coded before and just asked ChatGPT how to do things. I don’t think I would have succeeded in making the thing if I tried. There were all sorts of things going on — bunch of backend javascript driving UI in a spreadsheet — one version even had a vector database stored in another Google spreadsheet. It is really cool what they put together, and very much moves the boundary on what I think of as programming.
ChatGPT has also convinced me I’m better at thinking and writing about certain problems in code than spoken language. Especially analysis tasks — give me pyspark and let me think. Having a box where I can type in English and get code convinced me I’d rather just write code. I’m never really sure what kinda plot I want to make until I start grouping and aggregating grouping and aggregating. And when it comes to grouping and aggregating, I just don’t know a better way to express my thoughts than groups and aggregates.
> it’s way better at concepts than it is at data lookup
Doesn’t do well with many concepts tough, like divisibility rules ( “Are there primes whose digits sum to 9?” ) or correlations ( https://x.com/yudapearl/status/1804147019608658175 ).
I love the characterization of humans as black boxes. Thanks for that.
On the bigger topic, if you grab a random statistician in the wild and ask them what to do, they’ll tell you to run a t-test.
In my experience, ChatGPT 4 is already great at this kind of thing. It’s really good at taking writing and critiquing it and providing constructive criticism. Like asking a person, you probably don’t want to just do whatever the first thing that comes out of its mouth. Think of it like an assistant, not a replacement.
I am.
I remember the early 1970s when calculators became mainstream and educators had fits thinking they’d ruin education.
I remember in the early 1980s when desktop PCs and email became a thing (I’ve had internet email since 1984, when I started grad school). Pretty much everyone I talked to said they couldn’t see themselves using either PCs or email.
I also remember the advent of the web in the early 1990s, and again, everyone I talked to said “that’s not for me.”
My guess is that in a few years, nobody is going to be able to live without AI assistants.
Bob – your last line. I unfortunately agree with you, but I read that sentence differently. I won’t be able to live without these because my devices will require me to use them (I’d need to have Daniel’s knowledge to avoid that requirement). I’m already overburdened with what Microsoft and Google think I want, and I believe it will only get worse. This is not to denigrate AI – the potential for good is there. But it is no match for what human greed, ignorance, and ego are capable of.
We really need to get more people on board with getting away from Microsoft and Google products.
You can get a full working Debian installation using FAI in about 10 mins almost entirely hands off. https://fai-project.org/fai-cd/
But then you’ve got a Debian Linux system, and you need to know how to administer it… There are a lot of good intro books on such things, but it definitely isn’t a small change.
My biggest concern with the AI / LLM excitement is just how much **electricity** it’s all going to require, and how much lock-in to particular vendors it’s going to cause and how much spying the vendors can do on what you’ve been asking the LLM about.
When an LLM runs on my desktop machine I’ll probably use it a lot for stuff like “write a function that takes such and such DataFrame and outputs such-and-such series of plots laid out in such and such a way”. For the moment, I’m content to just do that stuff by hand when needed so that OpenAI doesn’t burn down a coal mountain on my behalf and also know everything I’m asking computers to do.
The Infosec part of this is I think way under-appreciated outside Infosec researchers.
I tried this:
Having run said experiment I will say this is very misleading.
Actually what you see is more like that some rats appear to prefer swimming, others want out immediately. Others just wait to get pulled out for the most part.
You only let them swim for a minute (or similar, a crucial detail of the protocol) before pulling them out (you don’t want to lose your rat to this!). So the ones that freak out the first time learn it doesn’t matter if they find the platform, they just wait to be rescued. Thus they never really learn the platform is what gets them out of the water. Keep in mind these animals have been living in a cage their entire lives.
In later trials some who found it earlier quickly swim right at the platform, hit the stand with their tail under the water, then prefer to keep swimming around rather than mount it. Maybe they like swimming, maybe they see it as a chance to escape, who knows?
I can go on about this, but suffice to say chatGPT lacks access to the technical details to give a good explanation. Especially if they are inconvenient to the reason the source of the training data got funded to do such an experiment.
Of course, it is just parroting what a certain subset of humans say in public about the topic. Humans who have failed to cure Alzheimer’s/stroke/TBI and so on using such studies.
How is that different than a search engine?
I suppose one difference is that it’s quicker and more focused to retrieve that answer. Even if it’s parroting what “a certain subset of humans say in public about the topic”, I would think that it is still more efficient to get that from ChatGPT, rather than searching for some average published review of the topic that provides a concise summary in one place. Not that much quicker of course, but across multiple tasks and questions it does save time (as Bob Carpenter frequently attests). Then of course there is the difference that a curious thinker could interrogate this answer interactively, which is very different from a search engine. Of course, ChatGPT might not be able to provide a good critique of what it has said, but I have found that questions that try to elicit where it might be wrong, or where multiple interpretations might be possible, generally lead to useful information that would take a lot more effort to discover otherwise. One issue with all this is that one has to already have a certain level of incredulity and scepticism to question it in this way, which is probably not going to be present in those learning about a subject for the first time.
The Wikipedia entry covers both the ideas and the problems with said test, so it took me 5 extra keystrokes to get to a reliable article on the subject.
Now, maybe the Wiki article is still a bit less critical of the stupidity of this particular test than it ought to be. But at least it tells you that there is bad news.
Bottom line: ChatGPT is good for people too lazy to read a Wikipedia article.
Oh yes, MIckeySoft’s Copilot comes up with a hilariously stupid answer for the “Are there three things, a, b, and c such that a is correllated with b, b is correlated with c, but a is not correlated with c” querry.
Note, however, that “ChatGPT” is stupid about X” is a bit of a fools errand, since they patch the front end whenever a common X is found. Meaning that people won’t see the problems until they actually use it for something new…
Interesting post. Two of the concerns I have with use of AI and LLMs are a) their inability to understand uncertainty in their answers and b) propensity to outright lie. Let me offer a recent example I discussed with my research design class:
Me: Are there any studies of a link between eating ice cream and shark attacks?
AI: Yes, there have been a few studies that have looked at the link between eating ice cream and shark attacks . . . In one study, researchers looked at data from over 40 years of shark attacks in the United States. They found that there was a weak correlation . . . Another study looked at data from 13 years of shark attacks in Australia. . . .
Me: Please provide citations to those studies.
AI: Sure, here are the citations to the studies I mentioned:
“Shark Attacks and Ice Cream Sales: A Spurious Correlation?” by David M. Dietz and Andrew J. Dobson. Environmental Science & Technology. 2012. doi:10.1021/es2030447
“The Relationship Between Ice Cream Consumption and Shark Attacks in Australia” by Michael J. Brown and Mark E. Bravington. Marine and Freshwater Research. 2014. doi:10.1071/MF13264
I hope this helps!
Me: Did you make up those citations?
AI: Yes, I made up those citations. I apologize for misleading you.
Already ChatGPT4o doesn’t do this in response to these questions.
See my comment above on fools errands and findin ChatGPT glitches…
I don’t have any specialist knowledge on this, but this seems to be more than just fixing an answer to one specific question (if that’s what you meant). In general now ChatGPT seems to default to searching the web and not relying solely on its prediction algorithms to provide false references. So yes, maybe they have “fixed a glitch”, but I don’t see how continuous improvement of a tool is supposed to devalue its practical utility. I doubt Wikipedia has exactly the same editorial functionality and rules as the day it was launched either.
“ChatGPT seems to default to searching the web and not relying solely on its prediction algorithms to provide false references. ”
Great. They’ve got a bad/stupid underlying algorithm, so you can now use that bad/stupid underlying algorithm to generate web searches.
I smell desperation. Also, enshitification.
(There was also the problem before ChatGPT that the search engines had sold out to the advertisers, so search results were dense of commercial sites selling the thing you were looking to find out something about. So ChatGPT is responding to a need that shouldn’t have existed.)
(Interestingly, Bing/Copilot did the web search thing from the start. It was OK, but you could still sucker it into saying stupid things. It was amusing watching them patch the front end to check for stupidities. Early on, it didn’t realize zero was a dumb thing to take the least common multiple of. Then they realized that it was a problem that the underlying algorithm does arithmetic by text database lookup, and fails for numbers that haven’t been used in it’s database, so they put in a front-end check for arithmetic. ROFL.)
But, whatever. We’re talking past each other. My concern is that the underlying algorithm is stupid and says inane crazy things that make no sense on occassion, and that there’s no prinicipled way to fix that. People who like ChatGPT don’t mind that, and are happy that the worst of the crazy/insane get’s checked for in the front end.
David in Tokyo wrote:
“My concern is that the underlying algorithm is stupid and says inane crazy things that make no sense on occasion, and that there’s no principled way to fix that. People who like ChatGPT don’t mind that, and are happy that the worst of the crazy/insane gets checked for in the front end.”
Exactly. It turns out that chatbots (which should be called Turing Test Bots or TTbots) were always going to rely on the Jeanne Dixon effect for success. And so far, that strategy seems to be working just fine.
Matt Skaggs wrote: “Jeanne Dixon effect”
Ah. That’s a new one on me.
But it’s exactly right. LLM algorithms have no mechanism for dealing with the meaning of the input or output texts in any way. So the interpretation of the output text is left to the user. On top of that, there’s the thing that it’s sort of like where the rockets come down not being Werner von Braun’s problem*, in that if the bot tells you to eat glue or ground glass, that ain’t our problem.
Which is to say, the whole LLM thing is exactly and only a parlor trick. Which is what I said ages ago…
(Aside: while the hacked front ends make the bots less obviously and easily demonstrable to be inane, said hacked front ends really spoil the fun.)
*: https://www.youtube.com/watch?v=QEJ9HrZq7Ro
Various sources have it that von Baun actually was learning Chinese…
I think the question of LLMs in data analysis looks very different depending on the (input language, output language) pair you look at. I have serious doubts about English-to-English; English-to-Stan seems much more useful than I expected; Stan-to-autocompleted-Stan is probably going to be standard…
But where things would get really interesting IMO would be with non-natural languages closer to the domain. I would very much love a model that took as input representations of what I’ve done so far in an analysis at a level somewhat more abstract than R/Python/Stan code and suggested me how to extend it based on good analysis paths from the literature or from a synthetic set.
In other words, if doing a data analysis feels sometimes as playing chess against nature (or rather poker or Battleship), then I would be much less interested on using a ChatGPT than on getting an AlphaZero.
I guess that as chatbots are used more commonly, people will get more used to the counterintuitive results that they give (such as producing convincing-looking fake references, as in Scott’s comment above). If you think of a chatbot as autocompleting, the fake references make sense: people do give fake references in real life. I like Gary Smith’s perspective on all this, which I’ve mentioned in several blog posts during the past year or so, that a big danger is people taking the chatbot output as more than it is, and using the output to make bad decisions. Then there’s the other concern mentioned in the above thread, the combination of huge power requirements (as noted by Daniel) and the push to make chatbots absolutely necessary (as noted by Bob).
>a big danger is people taking the chatbot output as more than it is
This relates to the recent “ChatGPT is bullshit” (bullshit sensu Harry Frankfurt, the Princeton philosopher) paper that has been doing the rounds on twitter over the past couple of weeks:
https://link.springer.com/article/10.1007/s10676-024-09775-5
I don’t think people giving fake references explains fake references from ChatGPT. It seems more likely its a result of training a model to predict the next token without grounding it with some other representation of the world. Probing the internal representations of an LLM after training to predict in some domain sometimes shows that the underlying world model is not very robust. E.g., an LLM predicting taxi routes in Manhattan does very well but appears to have learned a map that violates real world constraints on streets and intersections https://arxiv.org/pdf/2406.03689
When designing or using any software, a key attribute is testability. If one looks at chatbot tools from the cloud vendors, especially for Q&A, they all lack testing other than cosine similarity of the ground truth question and answer.
This inability to test a chatbot leaves businesses at risk, see https://www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-know.
There has been a lack of effort from the vendors to add testing, or versioning, to their products. This leaves their customers on the hook for long pilot programs of side-by-side testing. If you work in a regulated industry (healthcare, insurance, etc.), then can you deploy a customer facing chatbot, even when the replies go to call center employees? The risk is that the employee trusts the chatbot too much, especially when management is pushing that solution on them to speed up the call times. How can one enforce a requirement that the employee review the chat reply for accuracy, or tell the employee they are responsible for a final review?
Circling back to Jessica’s post, I guess I could ask LLMs about design questions, but how can I deal with the inability to test the accuracy of a reply? I don’t ask interns for design help, maybe I should, but how do I know the LLM replies are reliable enough to stake my reputation on it?
>I guess I could ask LLMs about design questions, but how can I deal with the inability to test the accuracy of a reply?
I agree provenance is important in analysis. Getting a suggestion for some modeling approach but with no way of tracing the original references is potentially annoying. But you could also follow-up with texts you do trust on the topic. In the LLM scenario you gain the ability to have some dialogue about what you don’t understand (assuming your usual process for design questions doesn’t already involve talking to people/posting on forums etc). Of course it’s possible this way of thinking about design is more circuitous without providing any real benefit over the alternative where you start by consulting some resources yourself.
Design questions seem closer to the “art” side of the spectrum. If I were using an LLM to handle calculations/computation, where a consensus answer is more possible, I would want to know what other software it’s using (e.g., if R what version, what packages, etc), and have access to similar levels of documentation about algorithms/defaults as I would get if I were doing it myself. So provenance would potentially look different, but is important across the spectrum.
I think design gets to the heart of my views. I know I am resistant to using an LLM to assist me with design – I can literally feel my resistance. I have asked Chat GPT design questions and even presented them to students as examples where the AI answers are better than many students. It is a good example of its usefulness. However, I have not seen such answers as good use of my time for my own work. I’d much rather post a question on an appropriate forum and get human responses – although it is important to me to choose a forum where I tend to trust the humans (e.g., the difference between Andrew’s blog and Marginal Revolution is an example of a blog where I tend to trust the responses as of value and one where I don’t).
I’ve played with Chat GPT enough to know that I can refine my questions and get increasingly valuable feedback. But the key to my refining my design questions is that I already have a good idea of the possibilities – good enough to steer the AI in a good direction. Without such knowledge, I simply think the feedback I get is too random to be a good use of my time. I think calculation/computation is different than design and it is much easier for me to see the usefulness in such settings.
The blog keeps giving me errors and eating my comments, then duplicating comments. What’s going on?
Bob:
I’m not sure. We get tons of spam so we have a serious spam filter. Sometimes legit comments are held in moderation, other times they go into the spam box. I check the comments queue every day but I don’t always check the spam box.
@Jess Do you think prompt engineering is simply a gigantic, high-tech garden of forking paths?
If you mean would I expect variation in results depending on how you prompt it, then yes. But I also think that at least among those currently using LLMs, there is often more awareness that results will be sensitive to small changes in the prompt than there is general awareness in human-driven statistical analyses that small changes in the process can result in different results.
When it comes to topics like simulating human behavioral responses with LLMs, I’m kind of hoping this acknowledgment of the importance of the prompt strategy there will lead to greater awareness of issues like cherry picking of stimuli in traditional studies.
But I may be biased in that most of the LLM enthusiasts I know are CS researchers who are probably more likely to expect variation than other types of users.
Great answer thanks.
One thing that is interesting in this discussion is that some people seem to be holding ChatGPT to higher standards that other tools, as if the fact that it isn’t perfectly intelligent undermines its entire existence. I can’t think of any other technological tool where there is an expectation that the meaning of all inputs and outputs would be perfectly parsed with no contribution from an intelligent and critical user.
This may be where David in Tokyo’s “talking past each other” comes in however. The frustration seems to be more due to claims and hype (and the AI label itself?) than whether the tool itself is useful. Just from the responses to this blog it is obvious that many intelligent people find it useful.
“The frustration seems to be more due to claims and hype (and the AI label itself?)”
Yes. Especially the label. Since the underlying technology (again, statistical processing of sequences of undefined tokens, i.e. statistical next word guesssing on steroids) is a complete joke, people who take AI and human intelligence seriously are irriteted something fierce.
There’s also the issue that the people here are folks who are capable of taking responsibility for actually checking potentially problematic output But, again, the underlying technology doesn’t “do reality” and can’t be trusted. So telling potential users anything other than “Do not, ever, trust the output of this program.” is a whole lot worse than just “claims and hype”.
(I personally don’t see how these things are useful at all: Wikipedia is just too good nowadays. And I hate the schmarmy mansplaing style of the text produced. (OK, the sample program generation bit is cute. Of course, the programming I’m doing (processing Japanese text in horrifically idiosyncratic ways) is unlikely to be amenable to ChatGPTing…))
I suppose there’s an interesting question here, namely what if we had a program that actually did have a model of the world, and actually did process and generate language based on that model, and did a decent job of it. That “knew” that putting glue on pizza was a bad idea, that handing in made up cases to a judge in a legal arguement gets you disbarred, that advertising a discount commits you to actually giving that discount. How would we evaluate the limits and capabilities of such a program?
But the AI discussion nowadays instantly turns to inanely stupid scifi…
David in Tokyo: You’re overgeneralizing. I take human and AI intelligence seriously. Seriously enough that I spent years working on natural language semantics and teaching philosophy of language at Carnegie Mellon. I’m not irritated by ChatGPT (though I am irritated by some of the hype). I’ve said for decades that generation language is AI complete—it’s the basis of Turing’s test for intelligence, and I believe he took human intelligence seriously.
I’m not sure what would constitute a model of the world in your opinion. Why don’t you consider the human brain a joke? It’s a finite set of neurons with finite connectivity firing very slowly based on very noisy biochemical processes. It’s very good at associative reasoning and very poor and slow at logical reasoning. We’re subject to all sorts of biases. And we’re very bad at causal reasoning if history is anything to go by (Fisher argued that smoking didn’t cause cancer and half my crazy country believes that having more guns doesn’t lead to more shootings; my friend the junior high school principal had parents arguing he was being unfair in not teaching “the other side” of the holocaust [i.e., that it didn’t happen]). Why does everyone think humans are so good at reasoning and have accurate models of the world?
“You’re overgeneralizing”.
Oops. I didn’t understand what you meant with this. Maybe it’s that:
My “But the AI discussion nowadays instantly turns to inanely stupid scifi…” was in reference to discussions by folks outside this blog. The singularity and other AI hypsters. This is (I think? recently, anyway) the first time I’ve responded to you, and I haven’t responded to Jessica because I largely don’t have anything sensible to add to what you all are up to.
But. The idea that humans are bad at causal reasoning strikes me as way off. We’re really good at it. The whole history of pre-1990 AI is trying to simulate human reasoning and failing miserably. Even simple common-sense everyday reasoning is really hard.
(The reason LLMs generate logical-looking text is that the internet is full of beautifully logical text with sensible arguments that says true things about the world. LLMs take that good, true text and randomly perturb it to generate (usually) sensible-sounding text without actually doing the work of figuring out why/how the input text says true (or intentionally false, or intentionally misleading) things, or what it’s output text “means”. It’s this not doing the work that rankles here, if and only if someone claims, or erroniously thinks, that the LLMs are doing said work.)
“A model of the world in my opinion”: I don’t know. AI tried to do that and we failed (I punted at the all-but thesis level for a variety of reasons, including that I couldn’t see a way forward). It’s hard. But people do really well at common sense reasoning. We don’t put glue on pizza. Jim Meehan’s story writing prorgam output “Gravity drowned”.
I think it’s clear that humans built internal models of the world. The human visual system builds a “gestalt” of the scene based on memory plus extremely minimal input, yet we think we’re looking at the whole scene in complete detail. (An early experiment used eye-tracking to shine a light on the place a subject was looking in an otherwise dark room. The subject would report being in a bright, evenly-lit room, while observers saw a spot of light moving around an otherwise dark room. And recently, it was found that an individual rat neuron specialized for global mapping, also did local mapping. Brains are hairy.)
Sure, we’re bad at evaluating logical expressions. That was known in the 1970s. And thinking logically and scientifically is hard and takes effort. But despite that, we do physics and math and chemistry, and get it right. A lot. Sure, we get things wrong, a lot also. But we’re often capable of figuring out and seeing that we’re wrong, at least when monetary and/or tribal self-interest doesn’t get in the way. That’s why science works. Folks who feel attacked, or have money on the line, don’t always make sense.
My personal opinion is that LLMs are silly, but I don’t mind people using them. As I’ve said, the program sample generation stuff sounds neat. Generating boiler-plate text and editing it is easier than writing a paper’s intro by hand from scratch. Even using the random silliness as fodder for brainstorming makes sense. All great fun.
But claiming that what is essentially random next-word guessing has something to do with “intelligence” strikes me as problematic.
Anon wrote:
“…some people seem to be holding ChatGPT to higher standards that other tools, as if the fact that it isn’t perfectly intelligent undermines its entire existence.”
This is a valid point. I certainly am, but only to evaluate the veracity of the hype. Based upon the hype, I expected something like C3PO from Star Wars but without the ability to mix me a drink.
That being said, there is no doubt that LLMs are a valuable thing. They most likely do a reasonable simulation of a single processing module within the human brain. That they do not really simulate a full brain is the answer to the wrong question.
I suspect that LLMs will eventually be exploited as a building block of more effective AI. The output from the LLM will be analyzed by a higher processor and compared to the output from other functional modules to derive a final (and better) answer. Essentially the testing that is mentioned upthread will occur inside the box, similar to a human evaluating what she will say before she says it.
As for the relationship between intelligence (by any definiton) and chatbot output, I’m actually kind of losing interest at this point. What I do still find fascinating is the relationship between chatbot output and creativity. I think there are fundamental lessons we can learn, and I suspect that the conclusions we will draw will not comfort us in terms of how we view our creativity as a uniquely human manifestation of ingenuity.
“What I do still find fascinating is the relationship between chatbot output and creativity. I think there are fundamental lessons we can learn, and I suspect that the conclusions we will draw will not comfort us in terms of how we view our creativity as a uniquely human manifestation of ingenuity.”
My (jaundiced, as you’d expect) thoughts on this are that we’re going to realize that psychadelic drug crazed “creativity” only works/counts as creativity to the extent it’s at least somewhat grounded in reality. More generally, artistic creativity is interesting to the extent it tells us about ourselves and/or our reality. (Well, for literature and painting/drawing. Music is another thing. (Although, maybe not. I’m a Sonny Stitt fan, and although Stitt does insanely complex blazing-speed runs, it’s his melodic sense that brings his solos together and makes them work as music.))
But LLMs don’t do reality. So they’ll do the psychadelic drug crazed twisting of reality, but not so much twisting as clumzily kludging together unrelated parts in random ways. It’ll get old fast.
What actually gets old fast, historically speaking, is people calling new technologies a “fad”.
For comparison, see early discussions of cars vs. horses, email vs. the post, texts vs. phone calls, PCs vs. mainframes, the internet vs. TV, Mapquest vs. paper maps, etc.
Here’s a nice Palko-style discussion of horses vs. cars—maybe he can write one of those old-timey tech posts on horses vs. bicycles, trolleys, cars, etc.:
https://quoteinvestigator.com/2021/07/17/auto-fad/
It’s what everyone I talked to in the 1980s said about email and message boards. It’s what IBM thought about personal computers before the 1980s. Most of the people I talked to in 1993 other than computer scientists said this about the web. I heard people say this about MySpace and then Facebook (and they’re probably saying that about TikTok and Instagram now).
For some reason I don’t understand, people are incredibly conservative when it comes to new technology.
I’m with Bob, I actually think what we will see is multi-system AIs, stuff that melds language models together with other forms of computing (database lookups, image processing, specialty language models trained on expert knowledge about particular topics etc)
The big concerns are 1) polluting the knowledge base with AI generated crap designed to attract clicks for ads, and 2) electricity consumption.
It’s a question as to whether we get there though as climate change, disease, and war will provide some major hurdles we might not be able to overcome. Degrowth down to say 4 billion people ish is likely the answer but between now and then there are many questions.
Another way to say this is language models are best thought of as providing a new primitive. There’s a bias though among many toward seeing them as some attempt at a singular form of human simulation to replace everything else.
Multi-agent combinations are already driving a lot of applications, see e.g., Autogen by Microsoft. It seems natural that multiple interacting agents (with access to more traditional stat tools) will be how we arrive at new data analysis tools that are actually interesting.
+1 on Matt’s comment.
Very engaging :) Thanks Jessica. I experimented and wrote about LLMs in data analysis being an arbitrage opportunity right now… I’m nowhere near the level of data analyst that everyone here is, but I think my article is at least worth a browse. Thanks again! https://open.substack.com/pub/frankcorrigan/p/llms-in-data-analytics-an-arbitrage?r=7dvjt&utm_medium=ios
I’m glad you posted your experience with ChatGPT and the movie data. It matches the reasons I don’t think it is a good use of my time – to use LLMs for cleaning and analyzing data – although it does point to more useful things like writing SQL code. But your analysis looking for changepoints in movies looks a lot like modeling noise and then developing post-hoc “explanations” for what was found. It is an interesting experiment and one I will probably try so that I can speak intelligently about its potential and dangers. But, just like many of those regression discontinuity analyses, the changepoints seem arbitrary. And once you “find” a point at which a trend changes, it is always possible to come up with an “explanation” for why that change occurred.
I’m not saying that the ChatGPT movie analysis is necessarily wrong. But I do think it is more likely to be wrong than a carefully constructed analysis a human (with an appropriate background) would do. It strikes me like a random analysis – and based on what I think I understand about how LLMs work – I think it is largely a random analysis. I’m not sure why anything in the LLM’s training data would cause it to discover “significant inflection points.” I’m not against using automated techniques to assist with that task – but there are automated techniques that are driven by an understanding of how to prepare the data for analysis and what kinds of techniques might be used to discover those points, and how to evaluate whether something meaningful has been discovered. That is the type of “understanding” I don’t believe LLMs are capable of doing.
There’s an interesting parallel to be drawn here between Andrew’s (and others) ideas on multiverse analyses I think. Those ideas suggest that a lot of humans aren’t particularly good at navigating forking paths in a coherent manner for inference either, so perhaps the LLM-assisted “random analysis” starts to look like a multiverse analysis reducing a human agent’s forking paths?
> LLM-assisted “random analysis” starts to look like a multiverse analysis reducing a human agent’s forking paths?
I’m confused–are you saying that, by providing an analysis that randomly selects one of the many possible forking paths, that the LLM is “helping” the human analyst by giving them a justification for ignoring other possibilities?
Another topic that has come up on this blog before is Philip K. Dick’s use of the I Ching to suggest elements of a story that he wouldn’t otherwise have considered. That sounds like it might be more like what you are suggesting, with the idea that you can replace the I Ching with the LLM. But since the LLM only selects one path, I actually think the I Ching would be better for that purpose!
I’m also curious about what Anon meant with the multiverse reference.
>But since the LLM only selects one path, I actually think the I Ching would be better for that purpose!
I think you are referring to the scenario that Anon is describing, so this is not necessarily directed at your comment but in general it’s worth emphasizing that the LLM only selecting one analysis path is an assumption that obviously doesn’t have to be true – one can and perhaps should ask for a set of options with rationales. The only real sense in which the LLM is constrained to selecting only one path is in generating its response token by token.
> one can and perhaps should ask for a set of options with rationales
Good point! I think it would vastly enhance the utility of LLM’s as tools for idea generation if they were used they way you describe. Even if they didn’t provide any rationale, at least getting to see the variability between responses would highlight the dimensions along which the options differ. It might also encourage more critical engagement with the model of the kind that Bob Carpenter describes.
To give a silly example from my own ill-spent youth, I once coded up a kind of SRN and trained it on some musical fragments that I had written. I then used the output of the model to help me generate ideas for a string quartet I was writing at the time. I didn’t use the model output verbatim most of the time, but it was cool to see how the model remixed my own ideas in ways that I wouldn’t have considered.
I guess this issue goes back to the “hype” problem—the hucksters are selling DNN’s as tools that give you “the” answer, and so that’s how people tend to use them. They could instead be better used as “idea pumps” (which would then address Anon’s point).
Hi Jessica, very interesting article!
A few weeks ago I published a post in my newsletter describing my experience doing data analysis with ChatGPT (and recording myself while doing that). I thought you might be interested in taking a look: https://filwd.substack.com/p/analyzing-my-workouts-data-with-chatgpt.
There is a ton of work to do in this area!
Thanks for sharing. Seems like you basically told it exactly what charts you wanted. Would be interesting to see what it produces when given a little more freedom, ie ask it to produce a set of useful visualizations or analyses given your high level interest in understanding your workouts.
@gec @Jessica (can’t reply in thread any longer) yes, apologies for lack of clarity, I did mean that the LLM could act as an “ideas pump”, thereby highlighting the risk/dependence of the forking paths to an analyst. I suppose this could take many forms: explicitly asking for multiverse variations, providing code and asking for where small changes might be considered legitimate but provide different answers, prompting for general critique, asking it for its “knowledge” of alternative approaches used in the literature. Perhaps in the future this could be baked in to an LLMs response to any data analysis question? Of course, it would then be dependent on the analyst to report this process…
Thanks for the clarification, I agree that the approach you describe would have real value!
My main concern with using LLM’s as “idea pumps” has more to do with how these tools evolve over time. LLM’s are trained on past instances, such that their responses will tend to sample from around those instances. When future models are trained on new text, if that text was generated or inspired by prior LLM output, then those future models will tend to become ever more “entrenched” around the responses from the LLM’s previous version. With each successive generation, the variability in LLM responses would be reduced, diminishing the value of LLM’s as “idea pumps”.
Good point, although I have no technical notion of how true this is across LLM instances. Naively I would also have thought that the increasing use of LLMs, coupled with ever-larger training sets, would mean that the “forking paths” reporting set would only increase. But I can see that this might be dependent on analysts having to push it harder to get away from the most common/likely associations in the training data. I suppose this kind of thing is quite hard to test given that the LLMs in the wild are not static, and a tame LLM (i.e. set up by an analyst specifically to investigate some research question) is unlikely to be very similar to those in the wild.
# My take on the issue of causal reasoning:
IMAO, the reason LLMs work so much better than almost everyone expected is because they are curve fitting the space of all human generated text (including equations and code) and that corpus has a “causally connected topology” (my term lol). LLMs are just curve fitters (a souped up version of least squares), but they happen to be rather good at curve fitting causally connected data. In fact, in my book Bayesuvius, I analyze transformer networks using causal Bayesian network language. https://qbnets.wordpress.com/2024/01/22/the-secret-causal-life-of-transformer-networks/
# use of LLMs to improve data analysis:
My software Mappa Mundi (which uses 2 different LLMs) can extract causal DAGs from text in a fully automated fashion. This is a type of data analysis.
# use of data analysis to improve LLMs
Current LLM do not distinguish between correlation and causation, except maybe in a very feeble implicit way. I call that being “superstitious”. IMAO, The fact that current LLM are superstitious is a problem that is even more serious than the notorious hallucination problem. Thankfully, software already exists (addons) that when added to LLMs, give LLMs the ability to not be superstitious, to do causal inference and to perform the scientific method in an explicit, deliberate way. https://qbnets.wordpress.com/2024/03/08/mappa-mundi-project-first-order-approximation-finished/