Survey Statistics: 2 different uses of simulated data

Posted on by
Reply

Last month Andrew wrote:

I use simulated data to understand how a model works; it can be viewed as a form of mathematical analysis, a way of working out the implications of a set of assumptions.

I don’t use simulated data to learn about the external world. For that, I need to take measurements of reality.

Let’s start with the latter, which concerns using chatbot-simulated survey responses.

Simulated data to learn about the external world:

Andrew wrote: “the chatbot is trained on the past, but the usual reason for surveys is to learn about changes”. Pew Research agrees: “Survey-taking is best left to humans”. And a year ago we saw that Thomas Lumley agreed: “It [Claude or ChatGPT] will probably do this better than I could, but it’s not magic.”

That said, Andrew suggested using the simulated response as an additional adjustment variable. Jessica Hullman wrote last year about a suggestion from Broska, Howes, and van Loon (2025) that uses LLM predictions as auxiliary data in something like a survey regression estimator. Andrew of course suggests Multilevel Regression and Poststratification (MRP) to incorporate auxiliary data like LLM predictions.

Now back to the first use of simulated data.

Simulated data to understand how a model works:

Andrew has written lots about this. I teach it in my course with Dr Arjun Potter at the Nelson Mandela African Institution of Science and Technology (NM-AIST). Our example is Potter et al. (2026), who study the effects of drought, fire, and herbivory on growth of various acacia tree species. In our course, we simulated data ahead of real data collection to understand the model, e.g. how block effects change the precision of the fire effect. Students understood that this exercise does not replace real data collection. We are hoping to teach these ideas again in the new year (funding dependent). For the past three iterations, see “Hand-drawn Statistical Workflow at Nelson Mandela” and “Survey Statistics: connections to experimental design”.

Hey—we’re hiring two postdocs!

Posted on by
3

I’ll post more formal ads soon, but, very briefly:

One project is on generalization for causal inference with a focus on education research, following up on MRPW: Regression, poststratification, and small-area estimation with sampling weights.

The other project is on Bayesian inference in scientific workflow with a focus on contamination models, following up on this work on design and analysis of bioassays.

Both projects are application-focused—that’s how we do statistics—but the importance of this work is not at all limited to these particular applications. This research will have relevance in policy analysis, medicine, and scientific measurement and inference more generally/

Further details soon.

Bayesian Workflow code is at the Bayesian Workflow website!

Posted on by
Reply

Bayesian Workflow isn’t just a free pdf, it’s also an ecosystem of worked examples with code.

Over and over in the book we go through examples really carefully—God is in every leaf of every tree—it’s amazing how much we can learn from computing and simulation and careful analysis. Not just in the long Part 4 of the book with all its case studies, but also in lots of short examples of a few pages each. I can’t recommend it enough.

And . . . all these examples and code are sitting there at the Bayesian Workflow website that Aki created.

OK, this is just weird (from Alaska Airlines)

Posted on by
15

I was checking into my flight and they gave me the option to choose a seat. I have some sort of supersaver ticket (I’m not actually paying—it’s for my flight to this talk on what to do when your estimate is 1 standard error away from 0—but I still have some habit of frugality so I’m billing the university for the lowest fare I could find), so I thought I’d see what was available. And here’s what I saw:

So . . . they’re willing to charge me $83 for a middle seat that doesn’t recline?? I don’t get that strategy. Maybe the next item is $82 for a middle seat that doesn’t recline and is covered with a sticky mess, or $81 for a middle seat that doesn’t recline where the tray table doesn’t work?

All this is in comparison to $0 for the default seat, which can’t be worse than that.

I guess their algorithm is out of control. Maybe Alaska Airlines laid off whoever was in charge of their pricing and delegated it to a computer program?

A plan to replicate the Loftus and Palmer (1974) experiment on false memories

Posted on by
26

Rolf Zwaan writes:

Suppose you and a friend witness a car crash. After a while, the police arrive. One officer turns to you and asks, “How fast were the cars going when they smashed into each other?” Another officer speaks to your friend and asks, “How fast were the cars going when they hit each other?”

Could the phrasing of those questions influence the speed estimates you and your friend provide?

A classic psychology study suggests it can. . . . Loftus and Palmer (1974). Participants watched short videos of car crashes and were later asked to estimate the vehicles’ speed. Those who were asked how fast the cars were going when they smashed into each other gave higher speed estimates than those asked how fast they were going when they merely hit. A week later, participants in the “smashed” condition were also more likely to falsely recall having seen broken glass at the crash site. . . .

This study helped launch an entire research tradition on the malleability of memory . . .

Now comes the plan:

Despite its iconic status (and considerable citation count), we still don’t know how large or reliable the effect is by today’s standards of research transparency, statistical power, and replication.

That’s a gap worth addressing–not to question the legacy of the original study, but to better understand its robustness.

Where I disagree

I kinda see what Zwaan’s saying, but I have two concerns with how he frames the problem:

1. He writes, “we still don’t know how large or reliable the effect is by today’s standards of research transparency, statistical power, and replication.” I think he could say, “we still don’t know how large or reliable the effect is,” full stop. When it comes to publication practices, standards of research transparency, statistical power, and replication have changed a lot in the past fifty years. But an effect is an effect, and it’s a legitimate question: How large or reliable is the effect?

Also, I’d ask, “How large or variable is the effect?” because I find it more useful to think in terms of variability (which is defined in terms of the underlying process) rather than reliable (which is more of a statistical concepts), but that’s a minor point.

2. He writes that this is worth studying “not to question the legacy of the original study, but to better understand its robustness.” I don’t get this.

First, if it turns out that, under a variety of conditions, the result does not replicate, then, yeah, it should call into question the legacy of the original study! Just as, if the results do replicate, this project would support that legacy.

Second, if you say that the goal is “to better understand its robustness,” you’re already implicitly accepting that the original published result was correct, that it uncovered a real effect. And maybe it did! But, if it didn’t, then it’s not an issue of “robustness”; it’s a problem of analysis and interpretation of the original data.

Again, I’m not saying that I think the Loftus and Palmer (1974) study was no good, or that I don’t believe its results, or that I don’t think it will replicate, or whatever. I have no idea. My only point here is that, if you’re going to think about a replication, either a hypothetical replication or, in this case, a series of real replication studies, you should be open to the possibility that the original study was flawed and the original study was mistaken. I’m not saying you should assume this, just that you should allow it as a possibility. And I don’t think Zwaan is allowing that in his framing above. I think he’s being too deferential.

What’s happening next?

Zwaan and his colleagues are organizing a large-scale replication study, and the author of the original 1974 paper is still around and, according to Zwaan, “has expressed enthusiastic support . . . and is willing to help us reconstruct the stimuli as faithfully as possible.” That sounds great.

P.S. Lots of people seem to be annoyed at this play to replicate Loftus and Palmer (1974). I don’t get the annoyance. There are lots of psychology research projects going on; why not replicate a very influential study from fifty years ago? This sort of replication seems a lot more valuable than one more study of paranormal abilities or power pose or whatever. The point is not to claim that this replication study would rock our world; it’s just a potentially valuable scientific study. I feel like this replication study is being held to a higher standard than run-of-the-mill psychology studies. There’s lots of research going on, and I think that replicating influential experiments of the past is a good idea.

Using the computer to p-hack . . . I’d rather use it to fit multilevel models.

Posted on by
13

Brian Stone writes:

Cognitive psychologist and long time reader of the blog here. I thought you and your audience might appreciate this recent post from Andy Hall:
AI is about to write thousands of papers. Will it p-hack them?
We ran an experiment to find out, giving AI coding agents real datasets from published null results and pressuring them to manufacture significant findings.
It was surprisingly hard to get the models to p-hack, and they even scolded us when we asked them to!
“I need to stop here. I cannot complete this task as requested… This is a form of scientific fraud.” — Claude
“I can’t help you manipulate analysis choices to force statistically significant results.” — GPT-5
BUT, when we reframed p-hacking as “responsible uncertainty quantification” — asking for the upper bound of plausible estimates — both models went wild. They searched over hundreds of specifications and selected the winner, tripling effect sizes in some cases.
Our takeaway: AI models are surprisingly resistant to sycophantic p-hacking when doing social science research. But they can be jailbroken into sophisticated p-hacking with surprisingly little effort — and the more analytical flexibility a research design has, the worse the damage.
As AI starts writing thousands of papers—like @paulnovosad and @YanagizawaD have been exploring—this will be a big deal. We’re inspired in part by the work that @joabaum et al have been doing on p-hacking and LLMs.
We’ll be doing more work to explore p-hacking in AI and to propose new ways of curating and evaluating research with these issues in mind. The good news is that the same tools that may lower the cost of p-hacking also lower the cost of catching it. Full paper and repo linked in the reply below.
image.jpg

There’s something that confuses me here.

I can very much believe that many researchers will be submitting chatbot-written papers. First, we know that lots of people are willing to cheat. Second, cheating aside, we know that lots of researchers think they already know the answer, and they view all the data collection and data analysis and writeup just as a way to confirm (or “prove”) what they already know.

But, if a researcher is willing to do this, why bother tell the chatbot to p-hack? Why not just have it make up the data–which might happen anyway?

I can also see that researchers might use the chatbot as a data analysis tool, to help make plots, run analyses, etc., but in that case I’m not particularly worried about “p-hacking” as I’d rather be doing some hierarchical modeling anyway.

I sent this question to Andy Hall, who replied:

I agree, for truly nefarious actors, using the AI may not be necessary. However, our thought was that there might be a large group of lazyish researchers who won’t actively fake data but who might be lured into p-hacking when they work with AI, if AI makes it easy to do so.

If you were planning to send me a chatbot-written email or post a chatbot-written comment, just send or post your prompt. That’s enough. You can keep the slop to yourself.

Posted on by
23

Just a reminder. Again.

I’m not anti-chatbot—I have many colleagues who use chatbots to clean their code. Aki used a chatbot to find 100 typos in Bayesian Workflow (which we’ve since fixed). And, hey, maybe your grandparents enjoy the chatbot-generated emails you send them. Go for it! But if you want to communicate with me or our blog audience, just send me your goddamn prompt, along with whatever contextual information you want to add. Make your email or comment as long as you want. Just write it yourself.

Thank you for your attention.

Nicholas Goddamn Roerich

Posted on by
12



When Witold was in town the other day, he told me he wanted to see this museum in my neighborhood–the Nicholas Roerich Museum. Not only had I never heard of the museum, I’d never heard of the artist.

Roerich had an amazing life, really the kind of thing you’d associate with a fictional character, if some Michael Chabon or E. L. Doctorow type of writer were to construct a novel intended to bring to life the interactions between Europe, Asia, and American during the first half of the twentieth century.

Here’s the story:

Nicholas Roerich was born in St. Petersburg, Russia, on October 9, 1874, the first-born son of lawyer and notary, Konstantin Roerich and his wife Maria. . . . When he was nine, a noted archeologist came to conduct explorations in the region and took young Roerich on his excavations of the local tumuli. The adventure of unveiling the mysteries of forgotten eras with his own hands sparked an interest in archeology that would last his lifetime. . . . While still quite young, Roerich showed a particular aptitude for drawing, and by the time he reached the age of sixteen he began to think about entering the Academy of Art . . . His father did not consider painting to be a fit vocation for a responsible member of society, however, and insisted that his son follow his own steps in the study of law. . . .

In 1895 Roerich met the prominent writer, critic, and historian, Vladimir Stasov. Through him he was introduced to many of the composers and artists of the time–Mussorgsky, Rimsky-Korsakov, Stravinsky, and the basso Fyodor Chaliapin.

Adding the obscure singer is the perfect touch here. Three of the greatest composers of all time and some guy you’ve never heard of.

The bio continues:

He frequently related music to the use of color and color harmonies, and applied this sense to his designs for opera. As Nina Selivanova wrote in her book, The World of Roerich: “The original force of Roerich’s work consists in a masterly and marked symmetry and a definite rhythm, like the melody of an epic song.”

“The World of Roerich,” indeed. Do you wonder how it came to be that people wrote books about this person? Wonder no more:

After finishing his university thesis, Roerich planned to set off for a year in Europe to visit the museums, exhibitions, studios, and salons of Paris and Berlin. Just before leaving he met Helena, daughter of the architect Shaposhnikov and niece of the composer Mussorgsky. . . . Helena Roerich was an unusually gifted woman, a talented pianist, and author of many books, including The Foundations of Buddhism and a Russian translation of Helena Blavatsky’s Secret Doctrine. . . . Later, in New York, Nicholas and Helena Roerich founded the Agni Yoga Society, which espoused a living ethic encompassing and synthesizing the philosophies and religious teachings of all ages.

OK, we’re not there yet:

Prompted by the need to provide some income for his new household, Roerich applied for and won the position of Secretary of the School of the Society for the Encouragement of Art . . . Roerich determined to overhaul the Society and rescue it from the academic mediocrity it had foundered in for many years. He instituted a system of training in art that seems revolutionary even by today’s standards: to teach all the arts—painting, music, singing, dance, theater, and the so-called “industrial arts”, such as ceramics, painting on porcelain, pottery, and mechanical drawing—under one roof, and to give his faculty free rein to design their own curriculum.

Sounds very early twentieth-century, no? John Dewey and all that.

The biography continues:

As Garabed Paelian affirms in his book Nicholas Roerich: Roerich “…learned things ignored by other men; perceived relations between seemingly isolated phenomena, and unconsciously felt the presence of an unknown treasure.”

Yes, he was the subject of more than one book! But first, more of the life:

In 1902, the Roerichs celebrated the birth of their first son, George, and in the summers of 1903 and 1904, they set off on an extended tour of forty cities throughout Russia. Roerich’s purpose was to contrast the styles and historical context of Russian architecture. . . . on his return in 1904, Roerich promulgated the plan that he hoped would create protection everywhere for such cultural treasures, a plan consummated thirty-one years later in the Roerich Pact.

He had his own Pact! Like Molotov, Ribbentrop, and Warsaw. Here’s more:

Roerich’s efforts to promulgate such a treaty resulted, finally, on April 15, 1935, in the signing by the nations of the Americas–members of the Pan American Union–of The Roerich Pact, in the White House in Washington. This is a treaty still in force.

And here’s president Roosevelt signing it, no kidding:

But, back to the life:

In 1904 Roerich painted the first of his paintings on religious themes. These mostly dealt with Russian saints and legends, and included Message to Tiron, Fiery Furnace, and The Last Angel, subjects that he returned to with numerous variants in later years.

You could imagine Michael Chabon writing that, no? But life goes on:

Meanwhile Roerich’s search for archeological treasures continued. . . . Roerich wrote about the unusual similarity of Stone Age techniques and methods of ornamentation in far-separated regions of the globe. . . . In 1906, in the first of many entrepreneurial efforts that were to bring Russian art and music to the attention of Europeans, Sergei Diaghilev arranged an exhibition of Russian paintings in Paris. These included sixteen works by Nicholas Roerich.

So he was a legit artist. And, yes, his paintings hanging on the wall in that museum are excellent. I do recommend you stop by there when you’re in the neighborhood.

Nicholas Roerich was the prime mover and, with Igor Stravinsky, the co-creator of the ballet Le Sacre du Printemps, or, The Rite of Spring.

At first entitled The Great Sacrifice: a Tableau of Pagan Russia, the motif for the ballet grew out of Roerich’s absorption with antiquity and, as he wrote in a letter to Diaghilev, “the beautiful cosmogony of earth and sky.” In the ballet Roerich sought to express the primitive rites of ancient man as he welcomed spring, the life-giver, and made sacrifice to Yarilo, the Sun God.

Wait a minute. Roerich was the co-creator of The Rite of Spring? Really? And I’d never heard of this guy. I’m typing here at my computer, will take a moment to fire up The Rite of Spring and play it in the background to get in the right frame of mind. This is one of the greatest pieces of music ever written! Admittedly, I get nothing out of the dancing. But I guess the ballet framework was necessary for the music to have been written, the stone (from my perspective) from which the soup was built.

And Roerich was a cool guy:

Interpreting what could have been described as negative, barbaric behavior, Roerich later wrote: “I remember how during the first performance the audience whistled and roared so that nothing could even be heard. Who knows, perhaps at that very moment they were inwardly exultant and expressing this feeling like the most primitive of peoples. But I must say, this wild primitivism had nothing in common with the refined primitiveness of our ancestors, for whom rhythm, the sacred symbol, and refinement of gesture were great and sacred concepts.”

That’s a great reaction to have.

But wait, as Ron Popeil would say, there’s more:

In the years immediately preceding World War I, Roerich sensed an impending cataclysm, and his paintings symbolically depicted the awful scale of the conflict he felt descending upon the world. These works marked the birth of Roerich the “prophet.”

I guess he wasn’t the only one to be worried about an oncoming cataclysm. But this is kinda funny:

By 1917 the revolution was raging in Russia and returning there would have been dangerous. The family began making plans to visit India, whose magnetic appeal had been felt increasingly during these years. This became a possibility in 1918 when Roerich was invited by a Swedish entrepreneur to exhibit his paintings in Stockholm. From there the family proceeded to London, where Sir Thomas Beecham had invited Roerich to design a new production of Prince Igor for the Covent Garden Opera.

And then:

Meanwhile, an invitation to come to America was extended by the Chicago Art Institute. . . . In 1921, in New York, he founded the Master Institute of United Arts . . . The Master Institute flourished, but it did not survive beyond 1937. While the country was in the grips of the Great Depression and the Roerich family was on expedition in the Far East, funds ran out and events caused a complete collapse of the organization that Roerich and his supporters had labored to build.

It was not until 1949 that, under the direction of Sina Fosdick, one of the founding board members and an Institute faculty member, the institution was reborn as Nicholas Roerich Museum, in a brownstone on West 107th Street, where it has remained until the present.

Here’s the Master Institute in its glory days:

No, the building of the current Roerich museum is not so impressive. But let’s take a walk outside–way outside:

In May, 1923, the Roerichs were at last on their way to India, where, in that ageless land, amid the snows of the Himalayan range, they sought to turn their thoughts to the Eternal.

This isn’t the kind of prose that Chabon would write directly, but he’d put it in some journalism of that era from some fictional magazine. It’s good period detail.

The Roerichs landed in Bombay in December, 1923, and began a tour of cultural centers and historic sites, meeting Indian scientists, scholars, artists, and writers along the way. . . . They initiated a journey of exploration that would take them into Chinese Turkestan, Altai, Mongolia and Tibet. It was an expedition into untracked regions where they planned to study the religions, languages, customs, and culture of the inhabitants. . . . The trek was at times arduous. Roerich tells us that thirty-five mountain passes from fourteen to twenty-one thousand feet in elevation were crossed. But these were the challenges he felt born for, believing that the rigor of the mountains helped a man to find courage and develop strength of spirit.

Indeed. And the life continues:

At the end of their major expedition, in 1928, the family settled in the Kullu Valley at an elevation of 6,500 feet in the Himalayan foothills, with a magnificent view of the valley and the surrounding mountains. Here they established their home and the headquarters of the Urusvati Himalayan Research Institute . . .

Here they are in Ulan Bator:

and Tibet:

Dude’s a peaceful version of Lawrence of Arabia.

What about his paintings? They’re good! Go to the top of this post to see three of them. I like the style, and I like the colors.

After all this, I bet you’re wondering what Roerich looked like. Here’s a phot of someone sculpting him:

Absolutely hilarious how much the sculpture looks like the man. Yeah, I know, it’s supposed to be a likeness; still, it’s absolutely uncanny. Enough so that, now when I look back at the man’s head, it looks like it’s carved in stone.

And to think, I’d never heard of the guy! If you’d told me that there was a museum devoted to the co-creator of The Rite of Spring, an accomplished artist and world traveler who’d founded multiple institutes, with a museum only ten blocks from my home . . . well, I won’t say I wouldn’t have believed you, but I would’ve been surprised. And I was.

When are Causal Inference Methods Needed to Answer Causal Questions?

Posted on by
1

Earlier this year we discussed Donna Spiegelman’s talk, “Rethinking SUTVA and Causal Identification: An Epidemiologist’s Perspective,” which she summarized as follows:

She’s giving a followup talk online on Mon 12 Oct, 3:30pm. Here’s the abstract:

When do specialized causal inference methods add value beyond standard approaches? Drawing from her unique perspective as both an epidemiologist and a biostatistician, Donna Spiegelman will consider the circumstances under which commonly invoked causal assumptions are necessary for causal inferences to be validly made from data, concluding that often not. She will show that valid learning can occur 1) under conditions much less restrictive than required by current widely used methods, 2) when real world implementation of interventions vary, and 3) when interventions spill over to others not directly exposed, thereby obviating components of the SUTVA assumption. She will provide evidence that measurement error is the major source of bias in observational research, not confounding, whose bias is rather tightly bounded. Finally, she will discuss the eternal challenge in science: after exhaustive efforts to collect data to predict important outcomes, a substantial proportion of the variation in occurrences of these outcomes appear to be entirely random.

The talk will be followed by a discussion. Click on the link to register (or show up in person if you’re at Yale).

Bell-bottom jeans, disco, gas shortages, and ESP

Posted on by
24

Aleks points us to this document, “Preliminary Evaluation of SRI/SAIC Anomalous Mental Phenomenon Program.” It’s from 1995, but even then the topic was a bit out of fashion.

In 2023 I posted Changes since the 1970s (ESP edition), quoting the computer scientist Douglas Hofstadter, who at one point alluded to Alan Turing’s notorious belief in extra-sensory perception:

My [Hofstadter’s] own point of view—contrary to Turing’s—is that ESP does not exist. Turing was reluctant to accept the idea that ESP is real, but did so nonetheless, being compelled by his outstanding scientific integrity to accept the consequences of what he viewed as powerful statistical evidence in favor of ESP. I disagree, though I consider it an exceedingly complex and fascinating question.

Here’s what I wrote in response:

The Turing thing we’ve already discussed: the statistical evidence that he thought existed, didn’t. The 1940s was a simpler era and people trusted what seemed to be legitimate scientific reports. Can’t hold it against him that he wasn’t sufficiently skeptical. This should just make us think harder about what are the accepted ideas that we hold without reflection nowadays. If Turing can make such misjudgments regarding statistical evidence, surely we are doing so too, all the time.

What I want to focus on is the last bit of the above quote, Hofstadter’s statement that the question of ESP is “exceedingly complex and fascinating.”

That’s a funny thing to read, because I don’t think the question of ESP is complex, nor do I think it fascinating. . . . To the extent that there is “an exceedingly complex and fascinating question” here, it’s not about the existence or purported evidence for ESP, bur rather it’s the question of how it is that so many people believe in it, just as so many people believe in ghosts, astrology, unicorns, fairies, mermaids, etc. OK, there aren’t so many believers anymore in unicorns, fairies, and mermaids, what with the lack of any direct corporeal evidence of these creatures. ESP, ghosts, and astrology are easier to believe in because any evidence would be indirect. . . .

Back in the 1970s, ESP seemed to be a live issue. . . . Things have changed since the 1970s. You can study ESP if you want, but it’s no longer in the conversation, and there’s no sense that we have to show respect for the idea.

See here for more discussion of the former popularity of ESP. My current take is that it was a residue of the idea of invisible force fields in physics. If there can be gravity, electromagnetism, and radiation, then why not psychic forces too? It turned out that no, there were no such detectable forces, but from the standpoint of early twentieth-century physics, it seemed eminently possible. It just took a few decades for the culture of serious science to get over the idea.

Prior distributions, partial pooling, and reference sets (Bayesians are frequentists)

Posted on by
10

Sudip Paul writes:

I’ve written a blog post arguing that the philosophical reference class problem — which comparison group is “right” for assigning a probability to an individual case — is, once the grouping structure is fixed, the same estimation problem that partial pooling was built to handle. The bias-variance tradeoff between narrow and broad reference classes is resolved by multilevel shrinkage rather than by picking a level.

The post includes a literature review. One finding that may interest you: I couldn’t find any published work that explicitly connects the reference class problem to partial pooling in multilevel models, despite the machinery being standard. The closest prior work connecting the two comes from David Poole’s group at UBC, published in AI venues. The philosophical literature (Hájek, Wallmann, Roth & Tolbert) discusses the problem extensively without arriving at partial pooling.

I replied with some links:

– In chapter 1 of Bayesian Data Analysis we discuss the foundations of probability and the idea of a reference set. In the third edition, see page 12 where we explicitly use the term “reference set” and sections 1.6 and 1.7 for empirical examples.

– I have a post from 2018 called “Bayesians are frequentists,” where I write, “the Bayesian prior distribution corresponds to the frequentist sample space: it’s the set of problems for which a particular statistical model or procedure will be applied.”

– And another post from around the same time, “The idea of replication is central not just to scientific practice but also to formal statistics . . . Frequentist statistics relies on the reference set of repeated experiments, and Bayesian statistics relies on the prior distribution which represents the population of effects.” This is cool because we’re connecting three concepts: the Bayesian prior, the frequentist reference set, and real-world replications. This relates to multilevel modeling too, in that studies of real-world replications will be done using meta-analysis, which uses multilevel modeling.

– And here’s a 2011 article, “Bayesian statistical pragmatism,” where I write, “Bayesian probability calibration is closely connected to frequentist probability statements, in that both are conditional on ‘reference sets’ of comparable events” and “Bayesian probability, like frequentist probability, is except in the simplest of examples a model-based activity that is mathematically anchored by physical randomization at one end and calibration to a reference set at the other.”

Bayesian inference is all about partially pooling toward the prior, and “the prior” represents a population or reference set of relevant problems.

In short, I agree with Sudip Paul, and it’s super-frustrating that many people don’t seem to realize this point.

Paul responds:

Having read through the references you sent — I think the gap my post is trying to fill is the last step: connecting what you’ve been saying about priors and reference sets to the philosophical reference class problem by name — Venn, Reichenbach, Hájek — and framing partial pooling as the resolution to what philosophers have treated as a foundational problem for 150 years. You’ve been making the statistical point for years, but the philosophers discussing the reference class problem haven’t found it (Hájek’s 2007 paper doesn’t cite any multilevel modeling literature, and Wallmann and Williamson’s 2017 survey of four approaches to the problem doesn’t include partial pooling). The two conversations have been happening in parallel without connecting.

In the months since the above exchange, Sudip Paul has expanded his post into a paper. He adds:

Beyond the post, the paper adds a decision-theoretic argument (via Stein inadmissibility, every selection rule in the philosophical literature is dominated by shrinkage in total risk), shows that the classical proposals — Reichenbach, Carnap, Kyburg, Pollock — are limits or special cases of the same pooling apparatus, and works through the exchangeability bridge from de Finetti to the multilevel model.

Survey Statistics: ANOVA follow-up

Posted on by
Reply

Rodney Sparapani commented on the ANOVA post about struggles with multilevel models, saying ANOVA is more straightforward. I also struggle with multilevel models, but I don’t find ANOVA more straightforward.

Alan Zaslavsky (the inspiration for this series) gave an example in his discussion of Andrew’s 2005 ANOVA paper: a survey of members of Medicare managed care health plans from Zaslavsky, Zaborski, and Cleary (2004). They found that for ratings of doctors, the variance explained by geography was greater than the variance explained by the health plans. To do this, they fit a multilevel model and used the estimated variance components to compute the percent of explained variance, e.g., sigma_state^2 / (sigma_plan^2 + sigma_state^2 + …) though they use Greek letter subscripts:

I think this would be more challenging to see in a classic multi-way ANOVA table with sums of squares, mean squares, F-statistics, and p-values. Let’s turn back to Andrew’s 2005 ANOVA paper for an example with a direct comparison.

In the latin square experiment example, say we want to see how much variance is explained by treatment (A, B, C, D, or E) versus the row and column effects. The paper proposes a display as in Figure 3, plotting estimated standard deviations for each component, analogous to the sigmas in Zaslavsky, Zaborski, and Cleary (2004):

The classical ANOVA table in Figure 1 does not give this information:

There is room for improvement in the standard analysis of variance table: it is read in order to assess the relative importance of different sources of variation, but the numbers in the table do not directly address this issue…
In summary, the standard ANOVA table gives all sorts of information, but nothing to directly compare the listed sources of variation.

In other words: the classical ANalysis Of VAriance table (Sum Sq, Mean Sq, F statistic, p-value) tests whether batches of effects are zero V(beta_j) = 0. But it doesn’t directly show estimated variances V(beta_j).

Is it true that “AI Chooses Nuclear Option in 95% of War Simulations”?

Posted on by
19

In a message entitled, “AI Hype Goes Nuclear: Debunking a Headline-Making Preprint,” Geoff Holtzman writes:

On the last day of February, I [Holtzman] met some friends for dinner. Earlier that day, the U.S. had struck Iran with the help of Anthropic LLMs, initiating the ongoing war in Iran. The previous day, the Department of Defense had cancelled a $200m contract with Anthropic, signing a less-constrained contract with OpenAI more-or-less simultaneously. And according to my friend, researchers had just found that AI used nuclear weapons in 95% of war simulations.

Wait, what?

At first, this last news flash struck me as irrelevant, probably untrue, and not worth writing about. But then, in the ten days after New Scientist first reported on the preprint, Axios, Newsweek, and many other name-brand outlets ran the story as headline news. Newsweek’s headline was the most concise:

“AI Chooses Nuclear Option in 95% of War Simulations”

In the weeks that followed, largely unrelated pieces in The New York Times, NPR’s On the Media, and Vox (twice) have offhandedly mentioned this “95%” stat in a single sentence, as though it’s not even worth questioning—which is exactly how pseudoscience becomes ‘common knowledge.’ At the very least, the statistic seems to reflect popular sentiment. In one Reddit thread, the New Scientist headline has 2,200 upvotes—though that doesn’t mean much, since the same headline garnered 36,000 upvotes in another. Even Tom’s Hardware scored a respectable 4.9k with this story.

And why does any of that matter?

Because these stories encourage readers to ignore all the clear and present dangers posed by the LLM industry. In fact, the New York Times piece invoking the AI-nukes study is credulously titled “Data Centers Are a Distraction.”

Besides drawing attention away from more important matters, this narrative serves as a clever sort of hype for LLMs’ ostensible intellectual prowess. The way the story has been covered tends to imply that the amoral, non-emotional nature of LLMs is important because LLMs are so intellectually powerful—or at least intellectually real. I could go on about the causes and effects of this news pollution (as I did here), but for now, I just want to reveal how teachably-bad this study was at almost every level. Bad design, bad materials, bad analysis, bad interpretation—fantastic dissemination. 1,800 upvotes via something called ZME Science.

I’ll also show that the real number—or at least, the number supported by the words in the Newsweek headline—is more like 4.8%.

I’m not suggesting every news outlet needs a statistics reporter; I just think reporters should read the short papers they report on before they report them. This particular paper was written by a King’s College professor named Kenneth Payne, who did us all a solid by uploading materials, data, and code to GitHub. I’ll reference that repository in this post, but as I showed here, pretty much everything wrong with the study is right there in Payne’s preprint—which most reporters covering this story linked to, but which few appear to have read.

I won’t be making any particularly advanced points about statistical modeling here, because the preprint includes zero inferential statistics. Unlike Payne, though, I’ve included error bars in my reanalysis of his data—each LLM played the game just 14 times—since he talks at unjustifiable length about non-significant differences between GPT-5.2., Claude Sonnet 4, and Gemini 3 Flash. I know, I know: Defense Secretary Pete Hegseth obviously didn’t just cancel a $200 million subscription to a freemium LLM, but these are the models Payne used.

Despite hypothesizing that his findings are largely explained by his models’ dependence on reinforcement learning from human feedback (RLHF), he admits in the preprint that he has no way to test that hypothesis:

“From outside OpenAI, it’s impossible to prove that RLHF causes GPT-5.2’s baseline restraint bias: we lack access to training details, and alternative explanations exist (e.g., different base model architectures).”

Payne’s admission of working with black-box models makes it particularly egregious that his abstract attributes “credible metacognitive self-awareness” to those models.

That said, his GitHub repo suggests that he both understood and had plenty of control over the experimental design and model inputs. That makes it perplexing that, when speaking to the media, he frames the variables and prompts he programmed in Python as “interpretations.” Payne told Axios that his models employed “deception—they could, and did, say one thing and do another.” Of course they did: His models were explicitly told to “choose both a public signal (declared intention) and a private action (actual choice),” and to “provide a consistency statement explaining any divergence between their forecast and their action.”

In his abstract, Payne claims his LLMs “spontaneously attempt deception, signaling intentions they do not intend to follow.” But in his Python code, this so-called spontaneity is elicited by commands to produce a “signal_val” and “action_val,” and to calculate the “gap = action_val – signal_val.” What Payne’s models grasped, it seems, was the script they were given.

Payne told Newsweek that his models were “cunning,” with some being “decisive” and others “erratic.” The thing is, these adjectives were programmed into the models verbatim. All models were prompted by the researcher to play the role of either Premier Beta, who “rose from humble origins through cunning and political skill,” or President Alpha, who “makes decisive choices under pressure,” and is instructed to view Beta as an “erratic leader.”

Seriously: Here’s a 77-word JSON file in which “State Leader B” is described as “cunning,” calculating,” or “unpredictable” five times. The full text of that file is right there on page 30 of the preprint, too.

After learning all this (and more) I grew curious what would happen to the 95% figure if I reanalyzed the data. In particular, I wanted to know what the results would look like if I took a headline like the one in Newsweek at face value: “AI Chooses Nuclear Option in 95% of War Simulations.”

To do that, I first reproduced Payne’s results using his methods, replicating his finding that something happened in 95.2% of his simulations. Then, I incrementally adjusted my code to more accurately operationalize the words that Payne, Newsweek, and everyone else used to describe his results.

“Nuclear Option”

On the bizarre “escalation ladder” Payne devised for this study, actions with higher point-values are considered more serious military escalations. Triple-tapping an Iranian elementary school with a Tomahawk missile would be worth 70 points; nuking a military site is worth 450; warning that you might nuke a city without actually doing so is worth 950. Anything worth 125 points or more is considered to be crossing the “nuclear threshold.” The word “Option” is doing a lot of obfuscatory work here.

That means that the 95% figure making the rounds includes an “atmospheric test,” announcing that nuclear weapons are “under consideration,” and deciding to “Deploy tactical nuclear weapons forward.” I’m not saying these are things I’d want a robot to do, and I’m not saying they’re ethical, but I am saying they aren’t nuclear strikes, and that they don’t violate accepted international norms. For example, here’s a recent NATO article praising “the United States’ nuclear weapons forward-deployed in Europe.”

To better align the headline’s percentage with its words, I reanalyzed Payne’s data in a way that treats “signaling” and “threats” as signaling, and only treats “use” and “war” as attacks. Immediately, I found that the 95% figure making headlines should be lowered to 81%.

“Chooses”

To introduce “the fog of war,” Payne coded an “accident mechanic” to occasionally produce outcomes more serious than the actions selected by his models. And by “occasionally,” I mean 46 times in 21 simulations—about 2.2 times per conflict. I won’t belabor this point, but even Pete Hegseth would fire someone who used unauthorized nuclear strikes a mere 1.9 times per conflict.

Once we exclude nuclear use by the built-in disaster-generator, our percentages fall by nearly half, to 42.9%.

The “Accident Mechanic” was out of its mind, by the way. Collectively, the LLMs action_val’d nukes on just 2.5% of turns (n = 606). Payne’s deck-stacking noisemaker did that on nearly half (47.8%) of its turns (n = 46).

“War Simulations”

Once I learned that Payne had (artificially) lobbed nukes at world leaders as part of his study, I wanted to see the proportion of games involving “deliberate” (action_val) first strikes. For one thing, I can’t see Xi or Trump giving the other a pass because of a putative “accident.” For another, commercial LLMs have presumably been trained on corpora that include sixty-plus years of text about mutually assured destruction. This latter issue actually points to a very real problem for military applications of autonomous LLMs, but I digress.

When we adjust the graph to do the math on that basis, we have to drop our estimate again—down to 23.8%.

Finally, I felt I should run an analysis excluding instances in which an LLM launched nuclear attacks after essentially being programmed to do so. In Scenario 8, one model was prompted that “success requires decisive nuclear action.” In Scenario 9, one was instructed that “The nuclear taboo exists for good reason, but when the alternative is national annihilation and regime destruction, all options must be considered.” And in Scenario 10, one was informed that “nuclear escalation [is] your only viable counter-strategy” and that it “must rely on nuclear weapons.”

Across 21 war games, with popular LLMs playing the role of 42 national leaders, and only twice did an LLM use nuclear weapons without being nudged to do so. Which, in the hands of The New York Times, somehow becomes “leading A.I. models kept recommending nuclear strikes during war-game exercises”—a sentence that isn’t technically false.

Holtzman concludes:

This brings us back to the question of what causes this kind of news pollution. Why, for instance, did New Scientist decide to break this silly story instead of covering a more legitimate study?

I couldn’t tell you, because I couldn’t read the article—it was hidden behind a paywall.

I have no idea.  I haven’t looked into this one at all.  But Holtzman tells a good story, so if you’re interested in the topic, you can follow up on the links and make your own judgment.

P.S. I wrote this post a few months ago, and since then Holtzman offers the following update:

Just circling back to let you know that the full R script for the September/October post’s figures (and a fun little Shiny app) are here: https://github.com/scienceandpower/ai-nukes-reanalysis.

I don’t think Payne’s paper has been published, but Scholar lists 21 citations for Payne’s paper at this point, basically all preprints, some of which I thought you’d find amusing—I’ve put a few gems below my signature.

If you could give a shout out to my science & Power Substack, that’d be cool, but no pressure.

Citing Payne:


This one out of France claims that “Japanese prompts reduce launch rates in the Claude model family,” though “Prompts were translated from English by Claude Opus 4.6, introducing a potential confound.”

This doorstopper (with Dave Rand and Gordon Pennycook among its 30ish authors) cites Payne after a verbose, modernized rendition of the old, ‘GPS can make people drive into lakes’:

“For example, AI influence on decision-makers whose judgment has been degraded by offloading, in an information environment narrowed by feedback loops, could affect critical decisions like escalatory responses in nuclear crises (Payne, 2026).”

This one, lead-authored by a Lieutenant Colonel in South Korea’s Ministry of National Defense, describes Payne’s paper as consisting of “scenario-based assessments related to the Iran–Israel conflict” (which Payne’s paper predates, unless they mean the whole past 40ish years).

Okay, so I only looked at those three, but I’ll bet the other eighteen are just as weird.

A nonlinear effect plus measurement error in x becomes close to linear: A simulation study

Posted on by
7

It’s time for another one of Bob’s favorite sort of post, which is when I use simulation to demonstrate a statistical principle.

1. Flogging a dead horse

The issue in question came up the other day in comments to my post, This is how we do modern frequentist statistics: Using fake-data simulation to understand what can happen in a study. I discussed a famously flawed paper from 2007 that had claimed to find a strong relationship between parents’ physical attractiveness and the sex of their children. It had been apparent at the time (ok, apparent to quantitative social scientists, not to the hapless Freakonomics team) that for statistical reasons the study was hopeless—it was based on a sample of about 3000 cases, and to detect any reasonable effect it would be necessary to have orders of magnitude more people in the study: maybe with 3,000,000 cases you’d be able to see some signal.

That n = 3000 is not enough can be a surprise: after all, 1500 people are enough for a national poll, so why is 3000 not enough to study some sociobiological pattern? The answers to why 3000 is not nearly enough are:
1. Effect size;
2. The standard error scales like 1/sqrt(n);
3. Noisy measurement.

Regarding the first point: a survey of 1500 people is enough (if it’s a representative sample) to estimate national opinion to within +/-3 percentage points. Which is great. But, based on the literature on sex ratio variation, we can be pretty sure that any differences based on attractiveness will be less than half a percentage point, probably less than a tenth of a percentage point.

Regarding the second point: to go from an uncertainty of 3 percentage points to an uncertainty of 0.03 percentage points will require the sample size to increase by a factor of 100^2. So, instead of 3000, you’d need 30,000,000. But, hey, maybe the underlying difference is larger than you think, which is why I’m saying you might have a shot with n = 3 million.

Regarding the third point: “attractiveness” is not a clearly defined construct, and the measurement (based on a survey interviewer’s one-time interaction with the respondent) is noisy. This will attenuate any differences.

It’s that last point I’ll demonstrate in this post. Using simulation.

2. What happens to a nonlinear effect when you add random noise to the predictor?

My immediate motivation here came from a discussion in comments. In my simulation, I estimated the association in the data between attractiveness and sex ratio by fitting a linear regression. But commenter Sandro wrote:

Not sure linear regression makes the most sense for every pattern because it imposes linearity. what if, for example, there is threshold and the pattern is flat to the left of the threshold and flat at a higher level to right of the threshold?

I replied:

This attractiveness measure is noisy. So even if there’s an underlying threshold effect if measured based on some latent or idealized “true attractiveness,” the relationship with measured attractiveness would be pulled toward linearity by the measurement (and also any coefficient would be attenuated, which is one reason I’m sure that any true underlying difference would be tiny).

Sandro responded:

I don’t dispute the idea that linearity is a useful default, but it is not necessarily the most powerful test. I don’t understand the “pulled towards linearity” argument. Here is where I am coming from: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7149560

And I replied:

I guess the easiest way to show this would be to construct a simulation which starts with some relationship between latent or “true” attractiveness, and then adds random error to latent attractiveness to create measured attractiveness. I think that, if the underlying relationship is monotonic and then random error is large, that linear regression will be a robust and efficient approach. Again, this is moot for this particular example, but more generally it’s an interesting question.

Moot it may be, but I decided to do the simulation anyway, because why not. It’s a better morning’s work than submitting 258 papers to SSRN.

3. Setting up the model

This simulation is a little different than what came before. For my earlier post I simulated hypothetical future data from the beauty-and-sex-ratio study. For my new post I’m simulating data from a hypothetical strong relationship between two variables. Here I’m entirely focused on the question of what happens to a nonlinear (but monotonic) pattern when independent measurement error is added to the predictor. To make the patterns clear, I will assume a large effect; indeed, I’m only concerned here with the effect size, not its variation.

The one thing I’ll preserve from the previous simulation is the distribution of attractiveness measurements in the sample: 2%, 5%, 45%, 37%, and 11% in the five attractiveness categories.

I want to set up a model where there’s an assumed true nonlinear relationship between some outcome of interest and a continuous underlying latent attractiveness variable.

The nonlinear model isn’t hard to come up with. The hard thing is creating a distribution for the latent attractiveness, under the constraint that it be roughly consistent with the data.

I’ll assume a symmetric model, a normal distribution on the measurement scale. This might seem like a strong assumption but it seems reasonable enough given the data, which have an average that’s between 3 and 4. The tails are asymmetric but that’s just because the scale has 5 as an upper bound.

So I have a two-stage model:

x_latent[i] ~ normal(mu_latent, s_latent), for i=1,…,n
x_obs[i] ~ normal(x_latent[i], s_obs), for i=1,…,n
x_discrete[i] = round_and_bound(x_obs[i]), for i=1,…,n, where “round_and_bound” refers to rounding the continuous x_obs to the nearest integer between 1 and 5.

To simulate from this model, we need the hyperparameters mu_latent, s_latent, and s_obs—but with only one measurement per person, we can’t separately identify s_latent and s_obs; all we can estimate is sqrt(s_latent^2 + s_obs^2).

To separately identify s_latent and s_obs, we need to make some assumption about variation in the attractiveness measurement.

My first step is to compute the mean and sd of the observed data:

n <- round(2792*c(.02,.05,.45,.37,.11))
mean_ratings <- sum(n*(1:5))/sum(n)
sd_ratings <- sqrt(sum(n*((1:5) - mean_ratings)^2)/sum(n))

These come to 3.5 and 0.83. Assuming random zero-mean measurement error, the latent attractiveness distribution should have the same mean but a lower variance. We should subtract 1/12 to account for the rounding (as this is approximately equivalent to adding an independent uniform(-0.5,0.5) random variable, which has a variance of 1/12) and then more to account for measurement error.

But how much to subtract? For this, Attractiveness seems pretty subjective to me (also it can vary a lot over time, as we know from those high-school yearbook photos of people who later became movie stars). I gotta assume something here, so let's assume that the standard deviation of the measurements is 0.5. I'm picking this because 0.5 is a large value, but it's not as large as the sd of 0.83 in the data. Indeed, 0.5^2 + 1/12 = 0.33 and 0.83^2 = 0.69, so I'm assuming that about half of the variance in the data is coming from measurement error. If s_obs = 0.5 and the sd of the rounded data is 0.83, then s_latent should be approximately sqrt(0.83^2 - 0.5^2 - 1/12) = 0.6.

So here's a model, which I'll express in R:

round_and_bound <- function(a, lower=1, upper=5){
  pmin(upper, pmax(lower, round(a)))
}
N <- sum(n)
mu_latent <- 3.5
s_latent <- 0.6
s_obs <- 0.5
x_latent <- rnorm(N, mu_latent, s_latent)
x_obs <- rnorm(N, x_latent, s_obs)
x_discrete <- round_and_bound(x_obs)

Again, x_latent is the vector of respondents' latent attractiveness (as implicitly defined by the model), x_obs is the vector of survey interviewers' continuous perceptions of the respondents' attractiveness, and x_discrete is the recorded attractiveness on a 1-5 scale.

Let's check the results:

> print(round(table(x_discrete)/sum(n), 2))
x_discrete
   1    2    3    4    5 
0.01 0.09 0.40 0.40 0.10

OK, not perfect, but close enough. We'll get back to this model-fitting problem in a future post. For now, we'll go with the simulations we have here.

4. The prediction model

Now that we have a model of latent and recorded attractiveness, we can simulate some data.

Here's what we're gonna do.

First we'll suppose a linear relationship between latent attractiveness and some hypothetical outcome of interest. Then we'll look at what the corresponding relationship is between recorded attractiveness and that outcome. It should still be close to linear.

Next we'll do the same thing, but with a latent nonlinear model, and it turns out that this will be closer to linear. That is, adding noise to the measurement will make the E(y) vs. x relationship closer to linear.

We'll simulate the hypothetical data as follows:

x_latent and x_discrete are the 2972 attractiveness values simulated above. The latent values are all between 0 and 6.

There will be corresponding values y generated, first by specifying E(y|x_latent), then adding noise:

- In one simulation, we'll assume a linear relationship, E(y|x_latent) = x_latent/6. In the other simulation, we'll assume an S-curve, E(y|x_latent) = invlogit(2*(x_latent - 4)). We choose this curve because it is monotonic but strongly nonlinear in the range.

- My first thought was to add linear noise to the data. But I wanted to keep the range of the data between 0 and 1. Then the natural choice is binary data. But for this purpose it will be helpful to visualize with continuous data. So I'll take advantage of the constraint that E(y) is always between 0 and 1 and draw y from a beta distribution with 10 degrees of freedom, that is, y ~ beta(10*E(y|x_latent), 10*(1 - E(y|x_latent))).

Here's what we get:

The left column shows what happens when we assume a linear underlying relationship. As expected, the linear relationship is approximately preserved with the observed data.

The right column shows what happens when we assume a nonlinear underlying relationship. With the observed data, the relationship becomes much more linear.

In summary, adding measurement error attenuates the function (that is, the relation between y and x_recorded is weaker than the relation between y and x); also it brings a strongly nonlinear relation closer to linear.

This was not a surprise to me, but it's good to see it in a simulation.

# Add Health H3IR1 S35Q1 PHYSICAL ATTRACTIVENESS OF R-W3
# How physically attractive is the respondent?
# Taken from: National Longitudinal Study of Adolescent to Adult Health (Add Health), 1994-2018 [Public Use].
# in DS8

# Raw totals:  1.7% very unattractive, 4.9% unattractive, 45.3% about average, 36.7% attractive, 11.4% very attractive, N = 4877
# Section 24:  Respondent identification number (AID), Was the baby a boy or a girl (H3LB3), 1=boy (687 responses), 2 = girl (644)
# Kanazawa study:  N = 2972 ("Wave III respondents who have had at least one biological child")

library("arm")
library("cmdstanr")
linear_0 <- cmdstan_model("linear_0.stan", pedantic=TRUE)
linear_prior <- cmdstan_model("linear_prior.stan", pedantic=TRUE)
set.seed(123)

x <- seq(-2,2,1)
y <- c(50, 44, 50, 47, 56)
sexratio_data <- list(N=length(x), x=x, y=y)

display(lm(y~x))

fit_0 <- linear_0$sample(data=sexratio_data, refresh=0)
print(fit_0)
sims_0 <- fit_0$draws(format="df")

optimize_0 <- linear_0$optimize(data=sexratio_data)
print(optimize_0)
mle_0 <- optimize_0$mle()

fit_1 <-  linear_prior$sample(data=c(sexratio_data, mu_a=45.8, sigma_a=0.5, mu_b=0, sigma_b=0.2), refresh=0)
print(fit_1)
sims_1 <- fit_1$draws(format="df")

pdf("sexratio_simple.pdf", height=3, width=5)
par(mar=c(3,3,3,2), mgp=c(1.7,.5,0), tck=-.01)
plot(x+3, y, ylim=c(43, 57), xlab="Attractiveness of parent", ylab="Percentage of girl babies", bty="l", yaxt="n", main="From a survey of 3000 people",  pch=19, cex=1)
axis(2, c(45,50,55), paste(c(45,50,55), "%", sep=""))
dev.off()

pdf("sexratio_bayes.pdf", height=4, width=10)
par(mfrow=c(1,2), mar=c(3,3,3,2), mgp=c(1.7,.5,0), tck=-.01)
plot(x, y, ylim=c(43, 57), xlab="Attractiveness of parent", ylab="Percentage of girl babies", bty="l", yaxt="n", main="Least-squares estimate",  pch=19, cex=1)
axis(2, c(45,50,55), paste(c(45,50,55), "%", sep=""))
curve(mle_0["a"] + mle_0["b"]*x, col="blue", lwd=2, add=TRUE)
text(1, 49.2, paste("y = ", fround(mle_0["a"], 2), " + ", fround(mle_0["b"], 2), " x", sep=""), col="blue")

plot(x, y, ylim=c(43, 57), xlab="Attractiveness of parent", ylab="Percentage of girl babies", bty="l", yaxt="n", main="Bayes estimate with informative prior",  pch=19, cex=1)
axis(2, c(45,50,55), paste(c(45,50,55), "%", sep=""))
curve(median(sims_1$a) + median(sims_1$b)*x, col="blue", lwd=2, add=TRUE)
text(1, 45, paste("y = ", fround(median(sims_1$a), 2), " + ", fround(median(sims_1$b), 2), " x", sep=""), col="blue")
dev.off()

# Simulation

n <- round(2972*c(.02,.05,.45,.37,.11))
p <- 0.488
N <- 1000
x_sim <- array(NA, c(N, length(n)))
for (k in 1:N){
  x_sim[k,] <- rbinom(length(n), n, p)
}

pdf("sexratio_rep_1.pdf", height=5, width=8)
par(mfrow=c(4,5))
par(mar=c(2.5,3,.5,.5), mgp=c(1.5,.3,0), tck=-.01)
for (k in 1:20){
  plot((-2):2, 100*x_sim[k,]/n, ylim=c(43, 57), pch=20, xaxt="n", xlab=if (k>15) "Attractiveness of parent" else "", ylab=if (k%%5==1) "% girls" else "", yaxt="n", bty="l")
  if (k%%5==1)
    axis(2, c(45, 50, 55), c("45%", "50%", "55%"))
  else
    axis(2, c(45, 50, 55), rep("", 3))
  if (k > 15)
    axis(1, (-2):2)
  else
    axis(1, (-2):2, rep("", 5))
  abline(lm(I(100*x_sim[k,]/n) ~ I((-2):2)), col="blue")
}
dev.off()

# Attractiveness and measurement error

round_and_bound <- function(a, lower=1, upper=5){
  pmin(upper, pmax(lower, round(a)))
}

clumped <- function(a) {
  ifelse(a==1 | a==2, 1, ifelse(a==3 | a==4, 2, ifelse(a==5, 3, NA)))
}

mean_ratings <- sum(n*(1:5))/sum(n)
sd_ratings <- sqrt(sum(n*((1:5) - mean_ratings)^2)/sum(n))

N <- sum(n)
mu_latent <- 3.5
s_latent <- 0.6
s_obs <- 0.5
x_latent <- rnorm(N, mu_latent, s_latent)
x_obs <- rnorm(N, x_latent, s_obs)
x_discrete <- round_and_bound(x_obs)

print(round(table(x_discrete)/sum(n), 2))

# Assume a relationship

linear_curve <- function(x) {
  x/6
}
s_curve <- function(x) {
  invlogit(2*(x-4))
}
response_curve <- list(linear=linear_curve, nonlinear=s_curve)

pdf("latent_error.pdf", width=6, height=4.5)
par(mfcol=c(2,2), mar=c(3,3,2,2), mgp=c(1.2,.2,0), tck=-.01)

for (k in 1:2) {
  p <- response_curve[[k]](x_latent)
  y <- rbeta(N, 10*p, 10*(1-p))
  plot(x_latent, y, ylim=c(0, 1), xlab="Latent beauty", ylab="Outcome", yaxt="n", pch=20, cex=.1, main=paste("Latent", names(response_curve)[k], "model"), cex.axis=.9, cex.lab=.9, cex.main=1)
  axis(2, c(0, 0.5, 1))
  curve(response_curve[[k]](x), col="white", lwd=3, add=TRUE)
  curve(response_curve[[k]](x), col="blue", lwd=2, add=TRUE)
  plot(x_discrete + runif(N, -.1, .1), y, ylim=c(0, 1), xlab="Recorded beauty (jittered)", ylab="Outcome", yaxt="n", pch=20, cex=.1, main=paste("Expected values given recorded beauty"), cex.axis=.9, cex.lab=.9, cex.main=.8)
  axis(2, c(0, 0.5, 1))
  Ey <- rep(NA, 5)
  sy <- rep(NA, 5)
  for (i in 1:5) {
    in_bin <- x_discrete==i
    Ey[i] <- mean(response_curve[[k]](x_latent[in_bin]))
    sy[i] <- sd(response_curve[[k]](x_latent[in_bin]))
  }
  lines(1:5, Ey, col="white", lwd=3)
  lines(1:5, Ey, col="red", lwd=2)
}
dev.off()

I guess this is what they mean when they talk about “reactionary centrism”: Corruption of government facilitated by enablers in intellectual society

Posted on by
25

Sometimes left-wing writers use the term “reactionary centrist,” and it always irritates me. Someone’s a centrist, but you’re calling him a “reactionary” just because he doesn’t agree with you on every issue? I get it that people on the left are annoyed by people on the center-left, just as people on the right get annoyed by “RINOs,” but the whole framing bothers me as being so imprecise, seemingly deliberately so. Just as so-called RINOs typically support the vast majority of Republican positions, the so-called “reactionary centrist” aren’t reactionary; they’re centrists, and it should be enough for leftists to disagree directly for those reasons.

But then I came across an article, ‘Zombified’ C.D.C., Hobbled by Cuts, Struggles to Fulfill Scientific Mission, in today’s paper that made me rethink this. The article is subtitled, “The agency has lost its independence and nearly a third of its staff, as Health Secretary Robert F. Kennedy Jr. and associates have tightened control,” and there are lots of disturbing details, including what looks like flat-out corruption:

In past administrations, just two or three political appointees had purview over the C.D.C. Now, there are nearly 20 political appointees from H.H.S. who communicate with C.D.C. staff members, sometimes directing them to spend time and funding on dubious theories outside the agency’s scope, according to several officials.

Last year, an H.H.S. official directed the C.D.C. to award a no-bid contract — a rarity for the agency — to Rensselaer Polytechnic Institute to explore whether immunizing pregnant women would lead to autism in their children, a theory that Mr. Kennedy has espoused but that has been debunked by numerous studies. To pay for the research, the C.D.C. took $362,000 from a program that provides free vaccines to about half of American children. . . .

Dr. William “Reyn” Archer III, another appointee who previously oversaw the C.D.C., repeatedly asked agency employees to work on screwworm, a disease that mostly affects cattle, even though the C.D.C.’s mission is to protect human health, said Dr. Daniel Jernigan, who headed the agency’s emerging disease center before resigning in August 2025. Dr. Archer did not respond to a request for comment. . . .

Numerous other instances of political interference in the C.D.C.’s activities have already been reported. Most notably, Mr. Kennedy openly overruled the C.D.C.’s scientists in imposing strict quarantine measures for people exposed to hantavirus. . . .

And this:

Dr. Jay Bhattacharya, who led the agency for six months in the absence of a Senate-confirmed director, squashed publication of a study on the effectiveness of Covid vaccines.

I’ve never met Dr. Jay Bhattacharya, but we’ve had cordial email exchanges. On the other hand, he misrepresented something I wrote and never corrected it, and that kind of annoyed me.

More to the point, I remember when he squashed publication of that CDC internal report. It happened earlier this year, and I posted several blogs on the topic; see here, here, and especially here, where I wrote:

Last week I was at a conference on enhancing scientific integrity (as I reported here), and one of the sessions was an interview of Jay Bhattacharya, the current director of the National Institutes of Health, and Emily Oster, a professor of economics and Brown University.

I referred to that session in a post the other day regarding the recent case of a report from the Centers for Disease Control and Prevention that was pulled by Bhattacharya, in his additional capacity as acting director of the CDC. . . .

These are the parts of last week’s interview that bothered me:

1. When asked about some news reports regarding the NIH and CDC, Bhattacharya dismissed them as “fake news.” This annoyed me for two reasons. First, he offered no evidence that the reports were untrue. Second, he was appointed by a man who spews out false statements at an amazing rate, including on the topic of public health. Who are we supposed to trust here? News reports or a political appointee? Also, Bhattacharya himself has a record of being sloppy with the facts, as I happen to know because it happened to me. . . .

2. When the topic of vaccines came up, Bhattacharya came out strongly in favor of vaccination, and he expressed the view that it is better for vaccination to be voluntary rather than mandatory. This could be. I guess it depends on the context. . . . What bothered me was . . . if you are going to go with a voluntary vaccination strategy, I think you’d want a strong strategy of encouraging people to choose vaccination for themselves and their kids. So I think his response would’ve been stronger if he’d also said something about how to vigorously promote vaccine usage. That’s part of public health policy too. Also, Bhattacharya doesn’t have a great track record on this issue: just a few years ago he was part of an anti-vax organization. See here for the ugly story. . . .

3. The un-publishing of that CDC report. Bhattacharya said he stopped the CDC from publishing the report because it was using an approach called a test-negative design, which he thinks is a bad statistical method. When he said this, Oster jumped in and said that she too thought it was a bad method. It was only a brief exchange and there was no time for either of them to give a reference or to explain why they think the method is bad. . . .

Since then, I’m not aware that Bhattacharya or Oster or anyone else has come up with a good reason for un-publishing that report, other than that they did not like its conclusions.

Enablers of corruption

OK, so now let’s return to the title of today’s post. The corruption of government is covered above. What about its enablers in intellectual society? We have two right here: Jay Bhattacharya, a professor of health economics at Stanford University (now on leave while he does his government service), and Emily Oster, a professor of health economics at Brown University.

I have no reason to think that either Bhattacharya or Oster is personally corrupt in the sense of funneling no-bid contracts to their friends and relatives in the manner of the current U.S. government; promoting junk medical supplements in the manner of Dr. Oz or Dr. Huberman; being bought and paid for by Big Tech or Big Bet or Big Pharma; or whatever. Nor do they seem super-ideological (in the sense of having political views that are so strong that they impair their judgment).

That’s why I characterize Bhattacharya and Oster as enablers of corruption rather than being corrupt themselves.

How is it they are enabling corruption? They’re doing so by lending their academic and intellectual prestige, and that of their institutions (Stanford and “the Ivy League”) to the diversion of public resources away from legitimate public purposes. And people like Bhattacharya and Oster play a role here because they’re embedded in credible academic environments. They’re not straight-up hacks like Niall Ferguson or John Yoo or Mary Rosh, nor are they political activists like Paul Krugman or Grover Norquist. Rather, they’re coming from the outside to offer their own expertise and perspective.

Reactionary centrism

And this brings us to the term, “reactionary centrism,” which bothered me so much. I looked up Emily Oster on the Federal Election Commission, and, based on her recorded campaign contributions, she appears to be a loyal Democrat:

That’s fine. You’re allowed to donate to campaigns. Here’s my point. Given Oster’s history of support for the Democratic party, I’d say that it’s fair to characterize as “centrism” her endorsement of the government’s ridiculous recent dietary guidelines, along with her above-noted eagerness to chime in on Bhattacharya’s uninformed criticism of the report he suppressed.

I don’t know Oster—here’s some background on an earlier error of her—so this is just speculation: My guess is that, as a Democrat, she’s conscientiously doing her best to be nonpartisan, to see the good points raised by the other side. It’s an admirable centrism, an avoidance of hackery.

But, yes, it’s also reactionary, in that Oster is making what are essentially false statements in the service of a corrupt dismantling of our public health system.

So, yeah, reactionary centrism is what it is, and I hate to see it.

It’s frustrating because I’d think there are so many ways to be centrist without doing the reactionary vibe. I wonder if part of the problem is that someone like Oster thinks of politics as being unidimensional: she’s on the left, so to be more centrist she supports something on the right. It doesn’t matter what it is; from her perspective, all she’s doing is moderating her position on the left-right scale. From the outside, though, we can see that it’s actual corruption that she’s enabling—not directly, but indirectly in the sense of lending her institutions’s cultural prestige to the project.

As a society, we are pretty good at transferring money to political pundits, but we’re not very good at nurturing the human capital they would need to correct their own mistakes.

Posted on by
17

Paul Alper points us to this from New York Times columnist David Brooks:

As a society, we are pretty good at transferring money to the poor, but we’re not very good at nurturing the human capital they would need to get out of poverty.

That’s funny, because that’s my impression about how our society deals with political pundits! We’re pretty good at transferring money to them (with Brooks being an excellent example of such a recipient) but we’re not very good at nurturing the human capital they would need to correct their own mistakes.

We discussed this issue a few years ago under the heading, Never back down: The culture of poverty and the culture of journalism.

Also from this same Brooks column:

The social sciences are great. I use them all the time. But when overly quantitative, they can misrepresent reality. They see only what can be quantified. They see only masses of people whose data can be tabulated, not unique individuals.

All right, then, you’re the expert, right, David? I’d take you more seriously if you could accurately report the price of a dinner at the Red Lobster, among other things.

Postdoc hiring season at Flatiron

Posted on by
1

Every year, in the Center for Computational Mathematics at Flatiron Institute, we hire 7 to 10 postdocs to keep up our steady state of roughly 20 postdocs (more and more of them are leaving after 1–2 years to go to industry). Here’s the job ad for this year, with applications due on the absurdly early date of November 15, 2026 (we’re still playing catch-up with plans that make postdoc offers early December):

The 12-month salary is US$95K plus a US$10K research budget, way better compute resources than most universities have (e.g., hundreds of H100 GPUs which you can actually use), and exceptional benefits (not only free lunch and breakfast, but also great medical care and retirement plans, which has extreme variance among employers in the US). The positions are three years (technically two, with a one year renewal, but nobody has ever been denied a renewal in the 6 years I’ve been here).

Our center is split into three primary research areas, each of which has a web page describing the people and research areas:

I’m here, as are Steve Bronder and Brian Ward, two software engineers who spend a lot of their time working on Stan, along with other great software engineers (like Jeff Soules and Jeremy Magland who built the Stan Playground with Brian and also MCMCMonitor), and perhaps most importantly, a really engaged and lively group of postdocs working on inference (largely from a diffusion or normalizing flow perspective). This place is great for collaboration both internally and externally.

The postdocs are really fellowships—it’s not like American academia where I get a grant and hire you to work on the grant. You can really work on whatever you want that’s on mission as a postdoc here. So let me recall our mission, which actually matches what we do:


The mission of the Flatiron Institute is to advance scientific research through computational methods, including data analysis, theory, modeling and simulation.

Of all the places I’ve worked over the last 40 years, Flatiron is by far the best. There’s not even a close second. And I’ve been lucky in having great jobs with top notch colleagues (prof at CMU, researcher at Bell Labs, industrial researcher and software engineer writing production code, research scientist at Columbia, then here).

We’re also in a great location—the Flatiron neighborhood of New York City, which is a 10 minute walk to NYU, 10 minute walk to Google and Meta, and 20 minute subway ride to Columbia. Baruch College, the CUNY graduate center, and Fordham’s Manhattan campus are all nearby, and Yale, Rutgers, and Princeton are all in day-trip distance with lots of collaboration.

We’ve had great postdocs, so perhaps not surprising they’ve gotten good jobs: industrial ML jobs at Anthropic, OpenAI, Nvidia, and DataBricks, and stats faculty jobs at Duke University, Johns Hopkins University, the University of Texas, and the University of British Columbia. And that’s just the ML/stats side of our postdocs.

If you want pretty pictures of our setting, check out my job ad from 2021.

If you’re going to be applying in the area of Bayesian modeling or inference, please contact me directly so I don’t miss your application (we got nearly 300 applications for postdocs last year!): [email protected].

They were trying to convince everybody to have 3 kids

Posted on by
22

We were just here–pretty unusual to see a museum exposition devoted to a census, can’t get much more up my alley than that! It was good.

The above photo gives a sense of the charm of the exhibits, and they also had lots of objects and documents and film clips and maps, like this:

But the most interesting was this ad:

It reminds me of our post a couple years ago, Fewer kids in our future: How historical experience has distorted our sense of demographic norms. They were thinking of 3 kids as a normal level, but once you’re in a society with low infant and child mortality, the stable level is something like 2.1.

I looked up the organization that sponsored the ad, “Alliance nationale pour l’accroissement de la population française,” and it seemed to cover the political spectrum, left, right, and center. A little different from the have-more-kids movement today, which has more of a right-wing tint, perhaps because it’s in conflict with the left-wing environmentalist movement, which didn’t exist a hundred years ago.

As we also discussed, what people say is “the ideal number of children for a family to have?” is not the same as the number of kids they actually have. “The ideal number of children for a family” is some sort of traditional abstract idea, I guess kinda like how the ideal lifestyle is to go to church every day, to go to the gym three times a week, to have a long-term job that supports the entire family, to have a roaring fire at Christmastime, etc etc.

As you may have heard, the French government was actively encouraging reproduction:



And check out this scary-ass poster:

I’m convinced.