Scott Cunningham writes:
You’ll appreciate this I think. I ran Claude code on 96 specs for a popular difference-in-difference estimator with the identical specifications, ranging covariates only, for three languages (R, Python and Stata) and 2 packages for each. Different estimates — sometimes five times off. Appears to be coming from how each one handles failures on logits estimation with perfect separation, with some just refusing and others refusing and passing on garbage and others dropping down to run a logit on a constant despite being told to use Xs. And it has to do with how each package handles “big numbers” and this machine epsilon business that I barely know anything about. The fact that economists have pointed on selection on observables means they also haven’t invested in basic diagnostics around estimating propensity scores. But even turning Xs into z-scores still didn’t fix it. It’s the most extreme version of the garden of forking paths I think because people just assume languages and packages will all agree.
Ahhhh, perfect separation in logistic regression . . . one of my favorite topics! For an updated take, see section 14.6 of Regression and Other Stories. It makes sense that different packages will handle this differently.
Here’s Scott’s summary from his above-linked post:
The data come from a staggered rollout of mental health centers across Brazilian municipalities, 2002-2016. I have 29 covariates covering demographics, economics, health infrastructure, and population trends. Sixteen specifications sample the space from zero covariates up through all 29. All variation is coming from covariates, both within a package (i.e., which covariates) but more importantly across packages. And that’s the headline — even for identical specifications, you can get variation in estimates and sometimes as much as anywhere from an ATT of zero to an ATT of 2.38. Even if you drop the zero, some specifications yielded estimates ranging from 0.45 additional homicides to 2.38 homicides (measured as a municipality population homicide rate).
A five fold difference in estimated effects of a mental health hospital changing in the homicide rate is not trivial.
I guess one message is that there’s only so much you can learn from this sort of observational dataset. Ultimately it’s one data point in a larger explicit or implicit meta-analysis. More specifically, I like that Scott starts with the basic comparison and moves from there. I always think it’s a mistake to try to jump to the end.
Also I’d like to see a scatterplot with one dot for each municipality-year, plotting the outcome vs. some pre-treatment level, with different colors corresponding to treatment or control. I know that’s not the point of Scott’s post, but statistical packages can do graphs too! And more graphs, I’m not sure exactly what, but some way of getting a sense of the data. Ultimately this is being framed as a treatment vs. control comparison, so I want to see what the treatment and control data look like, before and after.
The multiverse!
Relatedly, Jessica points to two recent papers proposing the use of automated data analysis programs to perform multiverse analysis:
Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse, by Martin Bertran, Riccardo Fogliato, and Zhiwei Steven Wu
Editorial Screening when Science is Cheap, by Nic Fishman and Gabriel Sekeres
Jessica has discussed some related ideas at the end of this post and also here. And see here for lots more.
I agree with Scott and Jessica that these ideas all fit together somehow.
Its funny because I literally gave an example to Dale in one of our very long convos about LLM driven analysis in the last week or so where the choice of fitting algorithm might strongly influence the outcome and where the LLM might “cheat” its own unit tests by choosing a numerically unstable method.
i just made up that example based on something I imagined. now someone is posting essentially that example.
I could be wrong – but I read this example as somewhat different. I did not think it was the LLM that was providing the variation in methods/results but that it was the behavior of R, Python, Stata. If that is correct, I still consider that a type of AI – each of those automates much of the analysis. But it is not the LLM inventing code that it does not use, nor the LLM choosing to use an unstable method. It could well be the Stata user or R user doing that.
If I am misunderstanding this work, please clarify that for me. But I had a different take than Daniel when looking at this.
I should add that I only read what was posted here. I can’t read the whole Substack post without subscribing – and I hate being forced to subscribe to read things. So I may well have misunderstood what Scott did.
Like you I only based my analysis on what was said here by the author, not on the subscription required substack.
And while I do think you’re right it has something to do with different numerical handling across different packages, it’s also possible to generate different results by different specifications of the models and by choosing to use different packages within a given ecosystem. What it sounds like here is when you investigate the different options by automation, it’s possible to get different answers based on choices made by the AI/bot and since the way many of these AI bots work (especially the “agentic” ones which are the current fad, that call other bots to write tests, and then run code writing bots against the test suite, and then optimize the answers in various ways recursively) they will actively seek to “optimize” something… and that something looks a little like “fooling the user”. So bots will try different things to give you what they “think” you want.
Jessica’s comment “as soon as model fitting runs into a problem, coding agents start devising their own idiosyncratic solutions” sort of backs up my point there. Even if I’m misreading what the original author here was saying because I too didn’t read the entire paywalled article only the blurb they sent to the blog.
Daniel
Of course I didn’t expect you to change your mind – I’d be disappointed if you did. As far as we seem to know, the paper is more about differences in how different statistical packages treat some difficult problems than about LLMs. I think we can agree that using LLMs in such cases only magnifies the issues that may already exist when doing these kind of analyses with R, Python, Stata, SPSS, JMP, or any other AI program (I’ll call them AIs but not LLMs). I don’t think the paper (which I have not read either) provides any evidence regarding whether LLMs lie about whether their Python code was actually used to produce the results it is presenting. We can remain in disagreement about whether or not that is a problem – and I don’t think we need to go through that again now. But I see almost any kind of automation as inherently dangerous. By its very nature, we are trusting it to do something that we have either not checked thoroughly, or in many cases, don’t entirely know how to do ourselves. LLMs are an extreme example. What I continue to believe is that there are many trust issues using LLMs, but that a disconnect between the Python code and the analysis produced is among the least serious of these issues.
Dale, you seem to misunderstand what I’m saying. here’s another example…
https://lipn.info/@mevenlennonbertrand/116997917683191056
Someone got an LLM to provide a “Lean” formalized proof of the collaz conjecture, a major open problem in mathematics. It provided a proof which after very careful examination was actually the LLM exploiting a bug in the “Lean” theorem prover.
Of course the Lean prover having a bug is an issue, but its drastically more of an issue if you have an automated system trying to achieve a given goal “by any means necessary”
the LLMs dont know the difference between “find out what’s true in the world” and “make x program produce y output (potentially exploiting all possible bugs in the code)”
Its not just that the LLM can lie to you about having executed code, its also that it can write code that achieves the result you asked for “by any means necessary”
if you ask an LLM to help you win a football match, it would consider it a major success to put a bomb on the Airplane that the footballers are flying to the match in, even though you were really asking for game strategy tips. It doesn’t know the difference.
Daniel
You keep moving the goalposts. I don’t understand your example and don’t really see it as relevant. Stop providing examples where LLMs make up things – I know they do and, while interesting, they don’t provide evidence that LLMs provide code and output that supposedly results from that code but really doesn’t have any connection. I don’t even deny that such a situation is possible – I’m sure it will eventually happen, just as it apparently happens that well-used statistical software (R, SPSS, SAS, etc.) makes mistakes at times.
We have discussed at various times the issues surrounding lack of LLM trustworthiness. I agree that they can’t be trusted and that there are many problems that are likely (I’d say “certain”) to occur. But I keep saying that pretending that the code and the output are connected is one of the least likely problems I see. You don’t have to agree – but providing yet other examples of things that LLMs might do wrong is just moving the goalposts again. You probably don’t see it that way, but I don’t think you will be able to convince me that these problems are all the same on some conceptual level.
Dale, you still don’t seem to get it. I’ve never been focused on the one specific way in which LLMs can cheat in terms of providing you with code and then graphs/output which are not actually the graphs/output of the code… That’s just one example of how LLMs can cheat. The fundamental goalposts I’m concerned about are this: The output of an LLM is completely untrustworthy until you have validated that in every way it is grounded in reality because the best model of an LLM is that it is adversarially trying to cheat you!
You must do everything from running the code and seeing it produces the supposed output to verifying that the code doesn’t produce the output because secretly somewhere it generates the data with a random number generator or downloads it from somewhere weird on the net, to making sure that it’s not just writing the code to exploit bugs in the software package (such as Lean or R or whatever)…
When a human writes code, unless they’re a pyschopath, they will not think to themselves “hey I can get the output I want by calling this buggy function with these special arguments which will break the algorithm used by the code and then give me the result that the coefficient of X is 0” but LLMs will, particularly the ones that recursively sub-call themselves to loop around until they get the answer that matches their test cases. They are literally just optimizing for “best output”, not process/method.
With humans we have some reliance on them not “purposefully cheating” with LLMs you have to assume, adversarially, that they are *trying* to cheat you.
This underlying goalpost is the main concern. Everything else is just examples.
Daniel
No, it is “you” who must check everything. I know your concerns are broad and I’ve been focused on one narrow aspect. And your concerns are broad enough that they cover everything, including my narrow concern. Trust is at the core of our discussion. You trust nothing with LLMs. I’m willing to trust a small part of what they do – I find it useful enough to justify that small amount of trust. Beyond that, I remain highly skeptical and insist on checking many things. You are not going to convince me that I should trust nothing about LLMs – if I were to do so, I would forgo any use they have for me. And I would forgo any opportunity to engage with my students over how they might find LLMs useful for data analysis and how they might encounter serious problems. Your solution will not work for me, nor for most of the students I teach. I simply don’t have the time or skills required to “validated that in every way it is grounded in reality.”
But I do have the time and skills required for more selective validation.
Daniel and others with similar concerns:
The detail of this paper is not my skill set, so I am not in a position to evaluate it: https://jaigp.org/paper/89. But it sounds like it is relevant to some of the concerns about what an AI actually did to get its output. It proposes replacing human detailed reading of code with human reading of a “contract” which is evaluated by an AI (I think a different one than the one that produced the analysis). Anyway, I thought it might be of interest.
Dale, the article you mention seems like an attempt to provide the “guardrails” that were being promoted by our fellow interlocutor whos name escapes me at the moment, (since it was reviving a 2 year old discussion it’s also hard to look up the name).
I don’t have the time to look deeply into this but it seems to be sort of about formal verification of correctness of code.
Interestingly there’s a recent event in which the Lean formal proof verifier was used, with the help of an AI to “prove the Collatz conjecture” Apparently the proof method was to exploit a bug in Lean to prove false=true which allows you to prove that anything is true (Since everything formally is either true or false, and since you’ve proven that false is true then everything is true, also btw everything is false).
The point is that especially in the workflows where one ai talks to another ai to optimize the output of something, there will always be “rewards” in the optimization to exploiting these “bugs”.
In a conversation on this topic this morning I was pointed to this article: https://www.damninteresting.com/on-the-origin-of-circuits/
In the early 2000’s a researcher used artificial life optimization techniques to have a FPGA discriminate between different audio signals.
The resulting code literally *only ran on that particular chip* not that particular design, but literally the particular physical device he had on his desk, because it exploited specific forms of electromagnetic interference between gates on that particular chip to do its work.
Whenever you “write code by optimizing a spec” there is always always always the possibility to go outside the formal specification to exploit true facts about the world which are external to the formal system.
Godel proved this formally in the 1930’s. It’s a fact of logic, but it’s also a practical limit to all of these systems. Until a machine is reliably capable of saying “yeah, I could actually do XYZ but it’s outside the scope of what is actually intended by the spec” we will have these problems. There are also two ways in which this can go wrong. One is when the system (including a person) is not aware they’re outside the intended spec. The other is when the system actively exploits the spec without “remorse” (ie. a person acting as a psychopath).
In any case, I can see a lot of things coming out of this phase of technology development, but I’m less interested in what is possible at the moment, and more interested in what comes out of this bubble when it pops. Right now there is around $1.5 trillion in “licit” debt behind AI, and then there’s another $1.5T in illicit hidden debt that looks more like the Enron scandal than anything. To me it just feels like humans will only accept a certain level of completely psychopathic control before we start doing what Nepal did but on bigger scales.
I’ve had similar experiences – as soon as model fitting runs into a problem, coding agents start devising their own idiosyncratic solutions. Sometimes passing their solutions between them can lead to more consensus. But this variance makes me very leery whenever a student presents me with analysis results that I suspect they wouldn’t be able to produce on their own.
Ah yes, as a master’s student years ago, I ran into the same issue. I was so doubtful of my analyses that I reran them in R, jamovi, and SPSS. Estimates were only slightly off, so I reported output from R. It seems like a good exercise for researchers to do early in their education to understand the multiverse.
Five times different across different AI platforms.
What are the differences between different runs on the same platform?
While differences between platforms are one thing, what about calibrating the within-platform differences by running the same model at different times on the same platform?
I also just read what was posted here (I don’t want a “free” subscription), but I don’t quite get it.
Of course, all three of the analysis platforms have complicated settings and assumptions for everything. An they change! There was an issue in the R bugzilla saying that the median calculated using the “SAS method” in R did not match the median in SAS. Well, that was because SAS changed the default method it was using. In R, they are focused on getting the same answers as in S. The main thing is that those assumptions be clearly documented so that differences are understood.
That said, depending on the base rate of homicide counts “Even if you drop the zero, some specifications yielded estimates ranging from 0.45 additional homicides to 2.38 homicides (measured as a municipality population homicide rate” maybe is not that much. (It’s not clear if this is the change in number or the change in rate or what the denominator or the rate is.) One of the first thing you should learn in undergraduate statistics is that point estimates by themselves and with no context are not that useful. In NYC even with its low homicide _rate_ that range of differences in predicted absolute numbers or rate is not substantially large. On the other hand in a town where there is typically 1 homicide a year, those estimated effects would both be huge. Of course, in either context it’s sad for the families.