19 thoughts on “LLM-generated Stan case study on Galileo’s inclined plane experiment”

1. Funny Bob, the last couple weeks or so I’ve been literally running inclined plane rolling ball experiments with my 15 year old son. We have real world timing data for rolling various balls down inclined tracks at different angles, as well as dropping them in free fall from heights between 50cm and 300cm or so. We used symmetry arguments to say that the results should depend on the sine of the angle the plane is at (note that x degrees and 180-x degrees are the same experiment viewed from opposite sides of the track so results should be symmetric about 90 degrees). We have compared the extrapolation of the results from inclined planes to “rolling vertically down a surface” and he even had the insight that he could use a little disk fridge magnet to roll it down a magnetic board vertically. We can clearly see the difference between rolling and freefall, we also found that the total mechanical energy is a constant (kinetic plus potential) during freefall, and used that to derive an expression for the total energy of a rolling ball or disk.

   We aren’t using LLMs to generate any code, we’re thinking about the results and plotting things by hand, the goal isn’t to get the answer quickly, the goal is to learn how to think about physics, how to formulate our knowledge into words, how to explore and discover things by exploratory data analysis, and how Julia works at the syntax level.

   I just found it humorous that we were both thinking about Galileo type experiments the last few weeks.

   Reply ↓

   • Nice timing. We replicated Galileo’s result in my high school physics class, which along with wood shop and metal shop (grew up in Detroit when it was still a factory town), was my best class in school. The math was all clear enough that when we replicated the Galileo experiment, I wasn’t getting a very precise answer from my measurements, so I fudged them so I’d get the right answer. Teacher saw that a mile away because I didn’t add enough noise to make the measurements realistic! Statistics remains tricky!

     Reply ↓

     • Kids are tempted to do this because they have no framework for handling messy data. Well, I guess sometimes even professional scientists do it too. So, anyway I kept reassuring my son that we just take whatever it gives us… unless there’s an obvious error like he dropped his cell phone and couldn’t press the stop button or the ball falls off the track. Being able to take the data with its messy dispersion and do something with it is reassuring. (in our case so far, we’ve just been least-squares regressing calculated acceleration vs sin of angle and things like that)

       This thread is one of several showing some graphs we produced. I managed to rope John Carlos Baez into commenting on the whole thing

       https://mastodon.sdf.org/@dlakelan/116744694827773287

       Here’s another, there are several more

       https://mastodon.sdf.org/@dlakelan/116750041939033305

       My son’s been taking a break, but I’m gonna try to get him to do the integral to calculate the kinetic energy in a disk magnet and then in a solid sphere as they roll. It’s an integral he could do by hand, but we will do it in Maxima to learn how computer algebra works.

       Reply ↓

2. Bob:

   Regarding the footnote: the answer’s in this comment thread.

   Also, yeah, that’s absolutely amazing that the computer could create that entire case study, which really would deserve an A if it were a student project. Just amazing. As amazing in its own ways as technological marvels such as airplanes and plastic bags.

   Reply ↓

3. you have misspelled his name

   Reply ↓

   • Fixed; thanks.

     Reply ↓

4. When I last wandered by this comment section, one of my central claims was that LLMs primarily work by memorizing their training set. Asking an LLM to recreate one of the most famous physics problems, which has dozens of write-ups scattered about online, is not much of a refutation. The use of acceleration is odd, as the most common approach to a solution relies instead on balancing kinetic energy and potential energy, but a little digging brings up a paywalled article on Wired about the acceleration of an object on a ramp.

   The random number seed is also suspicious. 42 is cliche, but you don’t want to be accused of cherry-picking some value. The current year is a natural alternative. The big LLMs like ChatGPT and Claude have the current date embedded in their system prompt, thus if they’re going to pick a year we’d expect 2026. Conversely, if you’re primarily regurgitating training data, you’d expect to repeat a prior year rather than the current one. Claude’s choice here? 2024. It would be icing on the cake if I could find a QMD notebook with that same seed; here’s one of two I could track down.

   Finally, Claude’s Stan code isn’t correct. Merely applying a lower bound to a parameter isn’t enough in this context, the probability function should also have been truncated. Like this:

   // Weakly-informative priors (must remain positive!)
   g ~ normal(10, 5) T[0,];
   sigma ~ normal(0, 5) T[0,];

   That’s not a big deal for g, the mean is two sigma away from zero and thus more than 97.7% of samples drawn will be valid. Poor sigma, though, will have a 50% rejection rate. Personally, I’d have used gamma( 4, 0.4 ) and exponential( 0.2 ) instead.

   Reply ↓

   • Finally, Claude’s Stan code isn’t correct. Merely applying a lower bound to a parameter isn’t enough in this context, the probability function should also have been truncated. Like this:

     // Weakly-informative priors (must remain positive!)
     g ~ normal(10, 5) T[0,];
     sigma ~ normal(0, 5) T[0,];

     That’s not a big deal for g, the mean is two sigma away from zero and thus more than 97.7% of samples drawn will be valid. Poor sigma, though, will have a 50% rejection rate

     Is this true? I’m pretty sure the constraining coordinate transform takes care of that

     Reply ↓

   • Hi, HJ:

     The log truncation factor from adding T[0, ] is a constant because the parameter bundles (10, 5) and (0, 5) are constants. So you can drop it for MCMC. Check out the Stan User’s Guide for a lot of examples and the Reference Manual for a full mathematical description of what’s going on. If there were Stan model parameters (or functions of parameters) used as prior parameters for g or sigma, then we would need the truncation. This is a common source of error (not including when it’s necessary) and inefficiency (including truncation when it’s not necessary) in Stan code.

     I agree that the seed “42” is a cliché. Nothing special about the seed—feel free to change. You’re right that if you ask a frontier chatbot these days to generate “random numbers”, its distribution is far from random. That’s why everyone’s moving to agentic AI where it just calls a random number generator when it needs random numbers, just like us.

     Reply ↓

     • I did wince a bit after I hit “publish” on that comment, because only then did I realize it amounted to a constant offset and thus the sampler would work A-OK without it. I agree with you on that point.

       But this is a two-variable model being fed into one of the best HMC samplers out there. I did a quick bit of benchmarking, and adding the constraints added maybe one millisecond to the overall runtime. The sampler itself exits in about twenty milliseconds. The degree of inefficiency is negligible here.

       I double-checked the Stan User’s Guide, and it tells me to add the truncation limits in. Repeatedly. It even discusses truncated distributions before suggesting placing limits on parameter values. There is no discussion of shaving off a few milliseconds by omitting these limits, that I can find.

       If your argument here is that Claude is somehow capable of intelligence, it should have followed the recommendations of the Stan User Guide and either used distributions that respected the truncation limits (I benchmarked my suggested distributions, and at worst they may have added a millisecond to the runtime; the “inefficiency” excuse doesn’t work there, either), or used a truncated distribution. That it didn’t adds another bit of evidence that it’s just copy-pasting human-crafted code, rather than understanding what its doing.

       Your mention of how terrible these models are at random numbers is uncanny, I’ve actually spent a decent amount of time on that very question. A moment’s thought should convince you that they should be excellent at generating random quantities, much better than a human. That they don’t is quite damning. I’ve gone so far down this road that I can trick a good half-dozen LLMs into generating better-than-human randomness. Notably, this trick doesn’t involve changing the user prompt in any way.

       Reply ↓

       • You’ll see that the example you point to involves parameters mu and sigma in the normal distribution that’s being truncated. That’s exactly the point I was making—this is the case where you do need the truncation statement. You are also going to need it if you want normalized likelihoods out.

         One might alternatively argue that Claude understands what it’s doing and thus realized it could drop the truncation bounds.

         I would only say that Claude is intelligent if I could define what intelligence means in a way that doesn’t reference human beings. If you want to define “intelligence” as what humans do, then obviously the LLMs are not intelligent.

         However you want to describe what the LLMs are doing, it’s not copy-pasting human generated code. Either literally or conceptually. It doesn’t “memorize” things, it builds a model. The model may assign high probability to things that are repeated a lot, but it’s not like it has a database it goes and checks for matches. There is no human-generated code in existence for the things it’s generating for me. I can ask it to take a Stan program block and optimize it for efficiency and it can work out what all the matrix dimensions are, where to transpose them, and how to organize them.

   • When I last wandered by this comment section, one of my central claims was that LLMs primarily work by memorizing their training set

     Yes, look up double descent and grokking. The whole avoiding overfitting and bias variance tradeoff thing turned out to be an obstacle to progress.

     This is surprising, but why is it a negative?

     Reply ↓

     • Because it turns LLMs from something capable of intelligence into a search engine that consumes several orders of magnitude more power than existing designs. Because all those people who thought they were pouring their hearts out or flirting with LLMs were instead tricked by Eliza 2.0. Because we’re exploiting and traumatizing hundreds of thousands of Africans to train these LLMs.

       Reply ↓

       • You’ll have to try one of the CLI interfaces (eg claude code) and have it work on a project you would never otherwise have time to get to. There is a learning curve to interacting with this new kind of “creature” properly. Like if you only had experience with cats and then tried to treat a dog like a cat (or vice versa) its not going to work right.

         As for intelligence, I don’t know. But they can overcome obstacles to achieve goals. The government is now starting to get involved so you probably missed the golden age already.

       • I’ve always considered myself more in the camp of treating LLMs as tools and evaluating them on their usefulness. 2–3 years ago, you couldn’t work with them for 5 minutes without them tripping up, and I felt closer to skeptics like you HJ. The AGI/singularity techbros still feel like they are just blowing smoke, but folks like you now come across much more like ostriches with their heads in the sand, forever clinging onto whatever the LLMs can’t do instead of embracing the incredible value they do provide.

         I’ve been coding every day for decades. I haven’t actually written a single line of code for at least the last 6 months, but I’m shipping out more valuable software than ever. I’m now even tackling projects that have been sitting on my ‘someday’ list for years.

         Whether LLMs are memorizing or can’t actually think is mostly academic.

       • kj
         Thank you for this – it reflects the way I feel about things. I am not a coder and I am happy to let AI do the coding for me. The mistakes it makes (from my experience, at least) are not in the coding, but in understanding context and making choices. Those are things I think humans should be doing anyway, so having the AI help with providing choices works very well for me. If my job was to produce code, then I guess I’d worry about whether it was correct or efficient or production-ready. But for my analyses, I don’t ask it to code so whatever coding it does works – because it produces answers. I worry a little bit about whether the answers are correct (and there are ways to validate that without going through the code line by line), but I worry mostly about the questions. Did it answer the right question? Did it provide good choices? Did it make assumptions that I didn’t realize and don’t agree with? The accuracy of the coding is the least worrisome part of the process.

       • I don’t get these comments. Maybe you have some secret search engine up your sleeve, but I don’t know a search engine that can create a well-formatted and technically correct case study involving math, physics, and statistics that doesn’t exist anywhere on the web. Maybe you used a different Eliza 1 than the stupid, repetitive one that I used to teach back when I taught NLP?

         That article you point to is about porn filtering, which isn’t an AI-specific problem. It’s a problem that any application on the web at scale has. I have a friend who does this for YouTube for Google, but they live in California, not Africa.

       • Here’s an example from today. Claude Opus 4.8 is very good at Stan now. For all the folks who say these LLMs are not “intelligent”, can we agree to call what they do “magic”? Opus magicked the following answer for me, refactoring this block of Stan transformed parameters:

          matrix[J_region, 4] regional_mu;
          regional_mu[ , {1, 3}] = Z_macro * gamma;
          regional_mu[ , 2] = zero_vector;
          regional_mu[ , 4] = zero_vector;
         
          matrix[J_region, 4] r_region
             = regional_mu + rep_matrix(sd_region, J_region)' .* (L_Omega * z_std)';
         

         to this one:

         matrix[J_region, 4] r_region = diag_pre_multiply(sd_region, L_Omega * z_std)';
         r_region[ , {1, 3}] += Z_macro * gamma;
         

         after telling me to transpose the definition of z_std. I did give it the rest of the code so it had the definitions of all the variables.

         It’s pretty much inconceivable it found a place to “copy” this from existing code on the web. It explained step by step why it did things this way and why I shouldn’t worry about the residual 4×4 transpose because it’s all in-cache and zero autodiff overhead.

5. Cool stuff here.

   My only complaint is that it doesn’t derive the formulas that it is checking. It just kind of hand waves about getting them.

   It would be interesting to see something similar for other early physics developments, like deriving Kepler’s laws or something.

   Reply ↓