Ted Williams and Me

Posted on March 12, 2026 1:47 PM by Phil

This post is by Phil Price, not Andrew.

Every spring, my friend Sam and I work on a consulting project together. The project is a program evaluation: the State of California requires the company (our client) to hire a disinterested firm (that’s us) to evaluate the effectiveness of one of the company’s programs.

In previous years, the program has been a big part of the client’s revenue and they’ve cared a lot about the answers. But the client has expanded nationwide over the past year, they’ve got contracts with a lot more companies in a lot of states, and this particular program is suddenly (really suddenly!) only a small part of their business… so small, as a percentage, that they pretty much don’t care about the numbers we get this year, as they said (politely) in a phone call. The analysis has a lot of required elements that are kind of a hassle for the client and it wouldn’t surprise me if they don’t participate in this program at all next year.

Still, we were on the hook for the report so Sam and I worked on it just as we always do.

With a couple of days to go before our deadline for submitting the report, we were on Slack on a Saturday afternoon hashing out a few things. We had this exchange:

Me: I get it that they don’t want or need to mess with this nonsense anymore, I’m happy for them and that’s good news for the industry in general. But it’s not a great feeling that the client doesn’t care about our work! There’s a moderately famous John Updike piece about the retirement of Ted Williams, in which he says “For me, Williams is the classic ballplayer of the game on a hot August weekday, before a small crowd, when the only thing at stake is the tissue-thin difference between a thing done well and a thing done ill.”

Sam: So you are Ted Williams in this scenario?

Me: You and me both. I mean, why are we both doing our best (or something approximating our best) when nobody else cares?

Sam: Gotta have standards and as an independent consultant, standards/reputation are about all that matters. I think it comes from internal compass, but that in turn means we can do this type of work successfully. To answer your question, like Ted Williams, I think we are both worried about personally being associated with deficient work.

There’s no particular message here, I just have the impression that there are a few people who follow this blog who appreciate hearing stories from the world of statistical consulting every now and then.

This post is by Phil

CMU finally shows up in Epstein files

Posted on March 12, 2026 9:26 AM by Andrew

Someone I know at Carnegie Mellon University was complaining about the unfairness of it all. The Epstein files include so many crooks, grifters, and sleazeballs from all sorts of prominent universities: Harvard, sure, but also MIT, Stanford, Columbia (Dr. Oz, Richard “Axel” Foley, etc., not to mention those creeps at the dental school), Yale, Princeton, the University of California, the University of Maryland, Rutgers, USC, Arizona State, . . . but nothing from CMU:

Really a bad break for student journalists at the famed Pittsburgh arts and technical school, as they didn’t have the opportunities offered to the Daily Cal, the Columbia Spectator and Bwog, the Harvard Crimson, the Yale Daily News, the Arizona State Press, the UCSD Guardian, NYU News, the New School Free Press, The Diamondback (go Terps!), The Tech, The California Tech. . . you get the picture. Even The Dartmouth found an angle.

But Carnegie Mellon . . . bupkis.

Creepy-ass computer scientist Roger Schank is all over the Epstein files, and Schank did have a CMU connection, but he didn’t seem to have been using a cmu.edu email account so he didn’t come up in my initial search.

But then I thought of searching not for cmu.edu but for *carnegie mellon*, and 92 documents came up, including this one which included a New York Times op-ed by Dan “redhead” Ariely, which contains these juicy bits:

Let’s be honest. We all lie. . . .

We’re not awful, immoral people, yet almost all of us want to gain from cheating. . . .

The problem with power is that it comes with some nasty side effects. When you put people in a position of power, they very quickly assume that position and, whether intentionally or not, start to abuse it. . . . lying, it turns out, is very much a social disease. . . .

As the saying goes, it’s funny because it’s evil.

The Irrational One continues:

In a study that my colleagues and I ran at Carnegie Mellon University, we planted a fake participant who looked like either a fellow student (wearing Carnegie Mellon attire) or a student from a rival university (wearing a University of Pittsburgh sweatshirt). We then asked the plant to make clear that he was cheating. When the student was wearing the Carnegie Mellon sweatshirt, his behavior signaled to his peers that it was OK to cheat — and their cheating increased. But when he was wearing the Pittsburgh sweatshirt, his dishonesty made cheating appear less acceptable, and it thus decreased.

I was curious about this one so I did some googling and found this press release from 2009:

Bernard Madoff’s arrest and guilty plea for scamming investors out of billions of dollars may serve as a cautionary tale for some, but could actually increase dishonest behavior by others, says Professor Dan Ariely of Duke University.

In a study published in the current issue of Psychological Science, Ariely and colleagues found that social norms exert a strong influence that can override other factors in determining how people behave after they observe dishonesty. . . .

Ariely, working with Francesca Gino of the University of North Carolina at Chapel Hill’s Kenan-Flagler Business School and Shahar Ayal of Duke, conducted a series of experiments testing the conditions under which students would cheat.

The researchers gave groups of students from Carnegie Mellon University a series of math problems, $10 cash and a sheet on which to tally the number of problems they could solve in five minutes. In one session, the students were given no opportunity to cheat; they submitted their answer sheets to a proctor who tallied the scores and instructed the students to keep 50 cents for each correct answer, and return the balance of the $10 in an envelope.

In order to assess the number of students who would cheat, students in a second session were not monitored. They were asked to tally their own scores then shred their tally sheets, and pay themselves 50 cents for each correct answer. Students in this group reported solving 50 percent more questions than the monitored students.

“What we saw in this experiment is that participants inflated their scores in order to take home a bit more cash,” Gino said. . . .

Inflating their scores, huh? That behavior sounds kinda risky. Good thing they were just students and not, say, Ivy League professors, or this sort of cheating could’ve put their careers in jeopardy!

The press release continues:

Next, the team tested what would happen if the students knew somebody else had cheated. The researchers hired an actor who dressed in a plain white T-shirt and looked like a university student. Sixty seconds into the test, the actor stood up and said, “I solved everything. My envelope is empty. What should I do with it?” He was told he was free to leave if he did not have any money to return.

Finally, the team ran a slightly different version of the experiment in which the actor wore a University of Pittsburgh t-shirt to participate in the experiment conducted at Carnegie Mellon. Again, the actor declared he was finished and had no money to return after one minute of the test. . . .

When the actor wore a white T-shirt and thus appeared to be a Carnegie Mellon student, a quarter of the students followed suit and claimed to have correctly solved all of the problems. However, when the actor wore a University of Pittsburgh T-shirt that indicated he was not a member of the students’ community, only one student (3.6 percent) imitated this behavior and claimed to solve all of the problems correctly.

With an ominous-in-retrospect coda:

“This is a frightening example of just how easily our own behaviors can be swayed by our judgments of the people around us,” Ariely said.

In all seriousness, this makes me feel sorry for Gino. Ariely was a senior researcher on the project! He did his part to destroy her career by teaching her that cheating was ok and even possibly a necessary part of science. As the saying goes, “Dishonest behavior can transition to continuous ethical transgressions.”

Those darn T-shirts

OK, here’s something funny. In his New York Times op-ed, Ariely wrote that the confederate was wearing “Carnegie Mellon attire” or “a University of Pittsburgh sweatshirt.” But in the press release it says he was wearing “a white T-shirt or a University of Pittsburgh T-shirt.”

So, which was it, CMU attire or a white T-shirt?

And, which was it, a T-shirt or a sweatshirt?

This is incredibly trivial, but . . . you should remember the details of an experiment you conducted, right? It appeared in a top journal! Also, you’re writing a NYT op-ed, that’s one of the most visible things you can do, so you’re motivated to check the details.

Well, I can check, at least. I went on google and found the published research article, “Contagion and Differentiation in Unethical Behavior: The Effect of One Bad Apple on the Barrel.” Let’s scroll down to the research design:

Finally, in the two identity conditions . . . we hired a professional actor to be our confederate. . . . The only difference between the in-group-identity and the out-group-identity conditions was the T-shirt that the confederate was wearing. Because the study was conducted at Carnegie Mellon University, the confederate wore a plain T-shirt in the in-group-identity condition and a University of Pittsburgh T-shirt in the out-group-identity condition.

Y’know, CMU has a legendary drama department (too many famous alumni to list here), so it’s kind of insulting that they hired a professional actor rather than just getting a student to do the job. What were they thinking??

In any case, you can see that the CMU press release is faithful in its description of the published article, while Ariely’s NYT description was wrong. I guess he was so used to making things up and not being called on it, that he was true to form and followed this strategy when writing for the Times. It’s only been 14 years, maybe they can run a correction. But, no, I forgot, they don’t run corrections on op-eds–at least, not when I’ve asked them to. Again, the point here is not the details–who cares if this dude was wearing a T-shirt or a sweatshirt–but rather the fact that Ariely was just riffing, making it up as he went along.

And . . . hey, the above-linked Psychological Science paper featured the famous paper shredder that may or may not have ever existed! I hadn’t realized.

OK, the CMU connection isn’t too strong here, but the experiment–if it actually happened–was conducted on its campus. So that’s something. Does it merit a story in the Tartan? I don’t know. I’m on the fence on this one.

A movie mashup

You know those cross-franchise movie mashups they’re doing now? Batman vs. Spiderman? John Wick time-travels to the Jedi universe? Etc. I’m thinking this should be done in science too, for example an experiment featuring Wansink’s bottomless soup bowl, Ariely’s paper shredder, Bigfoot, and a unicorn, all being used to demonstrate the effectiveness of the California Math Framework. It could be published in PNAS or Psychological Science, covered by NPR, featured in Ted Talks, and appear in the next edition of Nudge. Trebles all round!

P.S. In case you’re wondering why I’m wasting pixels on this, see this comment. I laugh because that’s better than crying. But, yeah, this sort of thing is much worse. No comparison. It’s just that if I going on about the news, I’d be talking about nothing else. And that’s not where I have much to offer you. You can go to the newspaper for that.

Postdoctoral research position at Princeton on psychological processes related to time (such as impatience and present bias) and their causes and political consequences.

Posted on March 11, 2026 10:23 PM by Andrew

Markus Prior writes:

I have an opportunity to hire a postdoc to work on an interdisciplinary project about time preference and patience in politics. The three most recent postdocs on the project have all gone on to assistant professor positions and are all still involved in the research. I am particularly interested in candidates with experience in experimental political science, behavioral economics, or political/social psychology (esp memory). The description of the position is copied below and listed here.

OK, by “behavioral economics,” I think he means “cognitive psychology,” but I get the idea.

From the project description:

The School of Public and International Affairs at Princeton University, invites applications for a Postdoctoral Research Associate or more senior research position. Applications are welcome from recent Ph.D.s and Ph.D. students who expect to earn their Ph.D. by the start date. The position will start on or about September 1, 2026.

The researcher will be asked to participate in the Time in Politics Project that focuses on psychological processes related to time (such as impatience and present bias) and their causes and political consequences. The individual will work on aspects of the project that may include analysis of existing survey datasets, development of original surveys, and experimental designs. The project will combine a variety of different research approaches. . . .

The successful candidate will have research experience in political psychology, behavioral economics, or cognitive/social psychology, and a strong background in quantitative research and statistical methods. Experience with questionnaire development and survey design software (e.g., Qualtrics, oTree) is highly desirable, as are data management and programming skills, preferably in STATA. (STATA will be used for this project, but proficiency in equivalent programming languages is acceptable as long as the candidate is willing to learn STATA enough to manage and extend the project files.) The candidate should be well organized, attend to detail, and respond to deadlines in a timely fashion.

OK, that last bit seems appropriate. If you’re gonna work on project about time preference and patience, you should be able to respond to deadlines in a timely fashion. I’ve actually never seen such a condition in a job description. Isn’t that assumed that if you take a job that you’ll do things on time?

Also kinda funny that they’re using Stata. Maybe the postdoc will be able to talk the PI’s into switching to Python or whatever.

In all seriousness, Prior has done and continues to do important work on political information on public opinion, so this sounds like a great project for someone with the right background and interests.

I wonder if the project will get into the debates regarding the effects of the economy on voting: to what extent are voters reacting to recent economic conditions, or reports of economic conditions, or something else?

More UFO bullshit in our science-media-industrial complex. (Yeah, there’s a reason this blog has a Zombies category.)

Posted on March 11, 2026 9:28 AM by Andrew

Recently we discussed the annoying thing where various members of prestige news media, people who are usually pretty strongly identified with the position that we should make our decisions based on hard empirical evidence, have this soft spot for the ridiculous theory that UFOs are space aliens. See also here. As always when this comes up, I’ll agree that all things are possible. UFOs could be space aliens, there could be ghosts in your attic, various people in the Bible may have indeed spoken directly with supernatural beings, Thor may really be up in the sky throwing thunderbolts, subliminal smiley faces actually could cause large changes in opinion on immigration, the moon landings might be fake, etc etc etc. I just haven’t any good evidence for these claims–and the evidence presented for UFOs being space aliens has that familiar pattern of disintegration upon close examination, and this motivates proponents of the idea to push hard on second-order arguments implying the suppression of evidence; see here for a discussion of one such example.

Here’s what I wrote when this came up before:

So . . . what happened? How did Scientific American, the Guardian, and NPR get conned? The answer is, they didn’t get conned. They wanted clicks, they got clicks. Many of the news media articles on this space aliens guy cover their butts, just a little, by pointing to skeptics. But not in a balanced way. The Scientific American article has three paragraphs about Loeb and only one paragraph about his skeptics, followed by an interview with Loeb featuring no tough questions. The Guardian article has a question mark in its title—good for them!—but the actual article, which is 16 paragraphs long, has only three and a half paragraphs expressing any skepticism. Etc.

Journalists have a reputation for skepticism. But, as we’ve discussed in the past in the context of Gladwell/Freakonomics/NPR/Ted/etc., credulity is a kind of superpower for a journalist. If you’re willing to believe, you can write these clickety-click stories with no pangs of conscience.

But not every journalist makes the decision to get conned:

How did sports site Defector nail it? I think it’s because, if you’re a sports journalist, you get familiar with stories where rich guys make stuff up and get fawning media coverage. From that perspective, this Harvard dude with the space aliens isn’t so different from some zillionaire who greases the local politicians and press in order to get funding for a new stadium.

The Nice House on the Lake

I hadn’t thought about these aliens for a few months, and then two new items arrived in the email to shake me out of my torpor.

Palko pointed me to this video from Rebecca Watson taking down “Avi Loeb, the Harvard Physicist Who Thinks It’s Always Aliens.” I don’t usually like videos, but this one has a transcript. Also it links to a video with the delightful title, “One Disgraced Scientist Gets an NBC Show, Another Becomes a Libel Bully.”

Those links are from a couple years ago so I don’t know why Palko pointed me to them now. In his email, Palko wrote, “I know you’ve done one post on Loeb but you might want to spend a bit more time on his case.” I find the story of Loeb himself to be super boring. A successful academic who doubles down on a speculative idea and soaks up the publicity? We’ve heard that one a few zillion times before. The interesting thing to me about these stories is not the deluded egomaniacs who associate themselves with them, so much as the otherwise-sane journalists who lap it all up. Hence my earlier posts on the topic.

Anyway, this dude seems to be in the news. I say this in light of this recent post by Paul Campos which amusingly ties this space-alien stuff to the stock market. Campos is writing a book on stupidity, and, as we know, the most interesting examples are when rich and well-connected people say or do stupid things (this one remains my favorite example), so I can see how this example caught his attention.

When Campos pointed me to his recent post, I replied:

Just a few days ago one of my regular tipsters recommended I post something on this, so now I guess I will . . . it should appear in Nov or so.

To which Campos responded:

You’re going to be really embarrassed if the aliens show up before then.

But I got in the last word:

Only if they reveal themselves. If they remain masked as humans, I’ll be just fine.

Maybe they’ll mask themselves as Harvard professors and elite journalists . . . who knows?

Survey Statistics: Blue Rose Research is hiring again !

Posted on March 10, 2026 4:00 PM by shira

As readers may know, I’m a survey statistician at Blue Rose Research. We survey the public to forecast elections and test political messages, used to advise Democrats. On this blog we’ve discussed our 2024 election retrospective and announced hiring (April 2025 and October 2025). And now we hiring again !

We are hiring a Data Scientist – Machine Learning (salary: $140k – $190k). This will be with our forecasting team, who track public opinion, forecast election results, develop resource allocation tools, and deliver polling analysis that clients rely on to set strategy. We’ve built a technical stack that enables cutting-edge statistical, machine learning, and engineering solutions.

Any additional roles we open up will be updated here. There’s an option to sign up for alerts for new roles via email if you’d like.

All positions are remote, with an option to be in person in New York City.

Please circulate and apply !

New course on generative AI for behavioral science

Posted on March 10, 2026 11:42 AM by Jessica Hullman

This is Jessica. It feels like an “old” course now that the quarter is almost over, but this winter at Northwestern I taught a grad seminar on Generative AI for Social Science. The goal was to survey emerging applications of generative AI (mostly language model agents) in the social sciences, with special attention to methodological and metascientific concerns that come up when AI is used to simulate or substitute for human observations or labels. I became interested in this topic last year as a result of the problems it presents to inference, but also the opportunities it may present to improve behavioral research, which we recently discussed here.

I joined up my computer science section of the course with a Communications section led by my colleague Aaron Shaw, which resulted in a good mix of students across AI and social science. This was great for discussion, as many of the Comm students were experts in survey methods or psychology, while the CS students were more knowledgeable about how transformers work and methods for probing their internals. We also organized a workshop on validating generative AI for social science in February, which some of the students attended and therefore got to see the authors of the papers they were reading present them live.

The only downside of mixing backgrounds was occasional friction with the more formal methods papers. A few times in class there were dismissive jokes made about the more statistically-demanding papers, I assume as a result of people finding the content challenging. The course was advertised as requiring grad-level experience in stats, so they should have known what they were getting into, but I think sometimes people read these things and assume it doesn’t apply to them. My stance on these things is typically that as long as you’re doing your best to understand the material, it’s not a problem and I’ll help you get through it. But I don’t have a lot of patience with the attitude that because it’s challenging to you, it’s ok not to try to understand it. Especially in a class about why we need to take seriously the challenges that LLM simulations present for drawing valid inferences about human behavior!

This week they present project proposals, which include things like exploring new prompting architectures based in cognitive theory, using ML interpretability methods to steer models in ways informed by social science, and studying belief elicitation and uncertainty expression in language models. It’s a good topic for a seminar course because there are lots of aspects of methods that haven’t yet been explored, and thus lots of opportunity to bring social science theories to bear on how we interact with language models, or to apply the latest methods in AI or stats to behavioral science questions.

Week 1: Course introduction: Can generative AI transform social science?

This week we set context, reviewing proposals that argue for the transformative power of generative AI for social science.

Horton, J. J. (2023). Large language models as simulated economic agents: What can we learn from homo silicus? (No. w31122). National Bureau of Economic Research. https://www.nber.org/papers/w31122.
Kozlowski, A. C., & Evans, J. (2025). Simulating subjects: The promise and peril of artificial intelligence stand-ins for social agents and interactions. Sociological Methods & Research. https://journals.sagepub.com/doi/full/10.1177/00491241251337316.
Bail, C. A. (2024). Can generative AI improve social science? Proceedings of the National Academy of Sciences, 121(21), e2314021121. https://www.pnas.org/doi/10.1073/pnas.2314021121.

Optional:

Dillion, D., Tandon, N., Gu, Y., & Gray, K. (2023). Can AI language models replace human participants? Trends in Cognitive Sciences, 27(7), 597–600. https://doi.org/10.1016/j.tics.2023.04.008.
Anthis, J. R., Liu, R., Richardson, S. M., Kozlowski, A. C., Koch, B., Brynjolfsson, E., Evans, J., & Bernstein, M. S. (2025). Position: LLM social simulations are a promising research method. Forty-second International Conference on Machine Learning Position Paper Track. https://openreview.net/pdf?id=cRBg1dtj7o.

Week 2: LLMs as surrogates I: Attitudes, opinions, social behavior

This week we start to read papers that evaluate how well LLMs can act as surrogates of humans. These readings focus on studies that use them to simulate human attitudes, opinions, and social behavior.

Park, J. S., O’Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023). Generative agents: Interactive simulacra of human behavior. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 1–22. https://doi.org/10.1145/3586183.3606763
Kim, Junsol, and Byungkyu Lee. 2024. “AI-Augmented surveys: Leveraging large language models and surveys for opinion prediction.” arXiv. https://doi.org/10.48550/arXiv.2305.09620.
Ashery, A. F., Aiello, L. M., & Baronchelli, A. (2025). Emergent social conventions and collective bias in LLM populations. Science Advances, 11(20), eadu9368. https://doi.org/10.1126/sciadv.adu9368 (very optional: you may be interested in this commentary/critique of the paper: https://doi.org/10.48550/arXiv.2505.23796).

Optional:

Chuang, Y.-S., Goyal, A., Harlalka, N., Suresh, S., Hawkins, R., Yang, S., Shah, D., Hu, J., & Rogers, T. (2024). Simulating opinion dynamics with networks of LLM-based agents. In K. Duh, H. Gomez, & S. Bethard (Eds.), Findings of the Association for Computational Linguistics: NAACL 2024 (pp. 3326–3346). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-naacl.211
Hansen, A. L., Horton, J. J., Kazinnik, S., Puzzello, D., & Zarifhonarvar, A. (2024). Simulating the survey of professional forecasters (SSRN Scholarly Paper No. 5066286). Social Science Research Network. https://doi.org/10.2139/ssrn.5066286
Park, J. S., Zou, C. Q., Shaw, A., Hill, B. M., Cai, C., Morris, M. R., Willer, R., Liang, P., & Bernstein, M. S. (2024). Generative agent simulations of 1,000 people (No. arXiv:2411.10109). arXiv. https://doi.org/10.48550/arXiv.2411.10109

Week 3: LLMs as surrogates II: Cognition and behavioral experiments

This week’s material expands on the first “surrogates” set to focus on using LLMs to simulate human cognition and experimental effects more directly.

Cui, Z., Li, N., & Zhou, H. (2025). A large-scale replication of scenario-based experiments in psychology and management using large language models. Nature Computational Science, 5(8), 627–634. https://doi.org/10.1038/s43588-025-00840-7.
Binz, M., Akata, E., Bethge, M., Brändle, F., Callaway, F., Coda-Forno, J., Dayan, P., Demircan, C., Eckstein, M. K., Éltető, N., Griffiths, T. L., Haridi, S., Jagadish, A. K., Ji-An, L., Kipnis, A., Kumar, S., Ludwig, T., Mathony, M., Mattar, M., … Schulz, E. (2025). A foundation model to predict and capture human cognition. Nature, 644(8078), 1002–1009. https://doi.org/10.1038/s41586-025-09215-4.
Tranchero, M., Brenninkmeijer, C.-F., Murugan, A., & Nagaraj, A. (2024). Theorizing with large language models (Working Paper No. 33033). National Bureau of Economic Research. https://doi.org/10.3386/w33033

Optional:

Chen, Y., Liu, T. X., Shan, Y., & Zhong, S. (2023). The emergence of economic rationality of GPT. Proceedings of the National Academy of Sciences, 120(51), e2316205120. https://doi.org/10.1073/pnas.2316205120
Ashokkumar, A., Hewitt, L., Ghezae, I., & Willer, R. (2025). Predicting results of social science experiments using large language models. Preprint. https://docsend.com/view/ity6yf2dansesucf
Peng, T., Gui, G., Merlau, D. J., Fan, G. J., Sliman, M. B., Brucks, M., Johnson, E. J., Morwitz, V., Althenayyan, A., Bellezza, S., Donati, D., Fong, H., Friedman, E., Guevara, A., Hussein, M., Jerath, K., Kogut, B., Kumar, A., Lane, K., … Toubia, O. (2025). A mega-study of digital twins reveals strengths, weaknesses and opportunities for further improvement (No. arXiv:2509.19088). arXiv. https://doi.org/10.48550/arXiv.2509.19088
Akata, E., Schulz, L., Coda-Forno, J., Oh, S. J., Bethge, M., & Schulz, E. (2025). Playing repeated games with large language models. Nature Human Behaviour, 9(7), 1380–1390. https://doi.org/10.1038/s41562-025-02172-y

Week 4: Bias, alienness, and other threats to generalization

When the goal is to learn about human behavior, relying on LLM simulations risks biasing downstream inferences. The readings survey ways that LLMs tend to misrepresent human response distributions and exhibit non-human-like errors, as well as metascientific concerns that arise from their availability as a cheap source of simulated data.

Wang, A., Morgenstern, J., & Dickerson, J. P. (2025). Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence, 7(3), 400–411. https://doi.org/10.1038/s42256-025-00986-z
Messeri, L., & Crockett, M. J. (2024). Artificial intelligence and illusions of understanding in scientific research. Nature, 627(8002), 49–58. https://doi.org/10.1038/s41586-024-07146-0
Westwood, S. J. (2025). The potential existential threat of large language models to online survey research. Proceedings of the National Academy of Sciences, 122(47), e2518075122. https://doi.org/10.1073/pnas.2518075122
Mancoridis, M., Weeks, B., Vafa, K., & Mullainathan, S. (2025). Potemkin understanding in large language models (No. arXiv:2506.21521). arXiv. https://doi.org/10.48550/arXiv.2506.21521

Optional:

Atari, M., Xue, M. J., Park, P. S., Blasi, D. E., & Henrich, J. (2023). Which humans? (No. 5b26t_v1). PsyArXiv. https://doi.org/10.31234/osf.io/5b26t
Dominguez-Olmedo, R., Hardt, M., & Mendler-Dünner, C. (2024). Questioning the survey responses of large language models. Proceedings of the 38th International Conference on Neural Information Processing Systems, 37, 45850–45878. https://dl.acm.org/doi/10.5555/3737916.3739374
Wang, P., Zou, H., Yan, Z., Guo, F., Sun, T., Xiao, Z., & Zhang, B. (2024). Not yet: Large language models cannot replace human respondents for psychometric research (No. rwy9b_v1). OSF Preprints. https://doi.org/10.31219/osf.io/rwy9b
Cummins, J. (2025). The threat of analytic flexibility in using large language models to simulate human data: A call to attention. arXiv preprint arXiv: https://doi.org/10.48550/arXiv.2509.13397.

Week 5: Validation I

This week we turn our attention to the methods that authors propose to use to check how well a language model simulates human behavior and draws inferences about the world, or to get valid estimates of their predictive accuracy.

Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337-351. https://doi.org/10.1017/pan.2023.2.
Manning, B. S., & Horton, J. J. (2025). General social agents (No. arXiv:2508.17407). arXiv. https://doi.org/10.48550/arXiv.2508.17407
Vafa, K., Chang, P. G., Rambachan, A., & Mullainathan, S. (2025). What has a foundation model found? Using inductive bias to probe for world models. Proceedings of the Forty-Second International Conference on Machine Learning (ICML 2025), PMLR 267. https://openreview.net/pdf?id=i9npQatSev.

Optional:

Neumann, T., De-Arteaga, M., & Fazelpour, S. (2025). Should you use LLMs to simulate opinions? Quality checks for early-stage deliberation (No. arXiv:2504.08954). arXiv. https://doi.org/10.48550/arXiv.2504.08954.
Larooij, M., & Törnberg, P. (2025). Do large language models solve the problems of agent-based modeling? A critical review of generative social simulations (No. arXiv:2504.03274). arXiv. https://doi.org/10.48550/arXiv.2504.03274
Aher, G. V., Arriaga, R. I., & Kalai, A. T. (2023). Using large language models to simulate multiple humans and replicate human subject studies. Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202, 337–371. https://proceedings.mlr.press/v202/aher23a.html

Week 6: Validation II

In contrast to heuristic approaches to validation that aim to show that LLM outputs are “close enough” to human ones, statistical approaches use some human observations to learn how to correct estimates drawn from LLM observations. These readings formally motivate why heuristic validation is not enough, introduce calibration frameworks, and present empirical results on how these methods compare to other repair strategies like fine-tuning.

Ludwig, J., Mullainathan, S., & Rambachan, A. (2025). Large language models: An applied econometric framework (Working Paper No. 33344). National Bureau of Economic Research. https://doi.org/10.3386/w33344
Broska, D., Howes, M., & Loon, A. van. (2025). The mixed subjects design: Treating large language models as potentially informative observations. Sociological Methods & Research, 54(1), 1074–1109. https://doi.org/10.1177/00491241251326865
Hullman, J., Broska, D., Sun, H., & Shaw, A. (2025). This human study did not involve human subjects: Validating LLMs as behavioral evidence. Preprint. PDF.
Krsteski, S., Russo, G., Chang, S., West, R., & Gligorić, K. (2025). Valid survey simulations with limited human data: The roles of prompting, fine-tuning, and rectification. arXiv preprint arXiv:2510.11408. https://doi.org/10.48550/arXiv.2510.11408.

Optional:

Song, Y., Kluger, D. M., Parikh, H., & Gu, T. (2026). Demystifying Prediction Powered Inference. arXiv:2601.20819.
Egami, N., Hinck, M., Stewart, B., & Wei, H. (2023). Using imperfect surrogates for downstream inference: Design-based supervised learning for social science applications of large language models. Advances in Neural Information Processing Systems, 36, 68589-68601. https://proceedings.neurips.cc/paper_files/paper/2023/hash/d862f7f5445255090de13b825b880d59-Abstract-Conference.html.
Hoffman, K., Salerno, S., Afiaz, A., Leek, J. T., & McCormick, T. H. (2024). Do we really even need data? (No. arXiv:2401.08702). arXiv. https://doi.org/10.48550/arXiv.2401.08702.
Zrnic, T., & Candès, E. J. (2024). Active statistical inference (No. arXiv:2403.03208). arXiv. https://doi.org/10.48550/arXiv.2403.03208

Week 7: AI as social scientist

So far we’ve mostly talked about LLMs being used as plug-in simulations for human data In surveys and experiments. This week we broaden to consider use of AI in other parts of the research process, like identifying what to research or what to manipulate in an experiment.

Manning, B. S., Zhu, K., & Horton, J. J. (2024). Automated social science: Language models as scientist and subjects (No. w32381). National Bureau of Economic Research. https://doi.org/10.3386/w32381.
Almaatouq, A., Griffiths, T. L., Suchow, J. W., Whiting, M. E., Evans, J., & Watts, D. J. (2024). Beyond playing 20 questions with nature: Integrative experiment design in the social and behavioral sciences. Behavioral and Brain Sciences, 47, e33. https://doi.org/10.1017/S0140525X22002874 (See also commentaries on this article).
Si, C., Yang, D., & Hashimoto, T. (2024). Can llms generate novel research ideas? A large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109.

Optional:

Musslick, S., Bartlett, L. K., Chandramouli, S. H., Dubova, M., Gobet, F., Griffiths, T. L., … & Holmes, W. R. (2025). Automating the practice of science: Opportunities, challenges, and implications. Proceedings of the National Academy of Sciences, 122(5), e2401238121.
Tong, S., Mao, K., Huang, Z., Zhao, Y., & Peng, K. (2024). Automating psychological hypothesis generation with AI: When large language models meet causal graph. Humanities and Social Sciences Communications, 11(1), 896. https://doi.org/10.1057/s41599-024-03407-5

Week 8: Causal discovery & explanation

Continuing with the theme of using AI to design experiments or support theory, this week we look at using models or interpretability methods to learn representations of stimuli or outputs to support causal inference, discover outcomes in text, and theorize about human behavior.

Imai, K., & Nakamura, K. (2025). Causal Representation Learning with Generative Artificial Intelligence: Application to Texts as Treatments. arXiv preprint arXiv:2410.00903
Modarressi, I., Spiess, J., & Venugopal, A. (2025). Causal inference on outcomes learned from text. arXiv preprint arXiv:2503.00725.
Zhu, J. Q., Xie, H., Arumugam, D., Wilson, R. C., & Griffiths, T. L. (2025). Using Reinforcement Learning to Train Large Language Models to Explain Human Decisions. arXiv preprint arXiv:2505.11614. https://doi.org/10.48550/arXiv.2505.11614.

Optional:

Tak, A. N., Banayeeanzade, A., Bolourani, A., Kian, M., Jia, R., & Gratch, J. (2025). Mechanistic interpretability of emotion inference in large language models. Findings of the Association for Computational Linguistics: ACL 2025 (pp. 13090–13120). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-acl.679.
Movva, R., Peng, K., Garg, N., Kleinberg, J., & Pierson, E. (2025). Sparse Autoencoders for Hypothesis Generation. Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:44997-45023. https://proceedings.mlr.press/v267/movva25a.html.
Zhu, J. Q., Peterson, J. C., Enke, B., & Griffiths, T. L. (2025). Capturing the complexity of human strategic decision-making with machine learning. Nature Human Behaviour, 1-7. https://doi.org/10.1038/s41562-025-02230-5.
Kim, J., Evans, J., & Schein, A. (2025). Linear representations of political perspective emerge in large language models. arXiv preprint arXiv:2503.02080. https://doi.org/10.48550/arXiv.2503.02080

Week 9: Belief-like representations and Bayesian inference

Behavioral scientists often take for granted that people have beliefs, attitudes, desires, and other mental states. This week we look at proposals for how to look for similar representations in language models. We also consider a Bayesian formulation of the prompting process that describes how the researcher’s expectations of reasonable data influence what they generate with language models.

Herrmann, D.A., Levinstein, B.A. Standards for Belief Representations in LLMs. Minds & Machines 35, 5 (2025). https://doi.org/10.1007/s11023-024-09709-6.
Yamin, Khurram, Jingjing Tang, Santiago Cortes-Gomez, Amit Sharma, Eric Horvitz, Bryan Wilder. (2026). Do LLMs Act Like Rational Agents? Measuring Belief Coherence in Probabilistic Decision Making. https://arxiv.org/abs/2602.06286.
Misra, S. (2025). Foundation Priors. arXiv preprint arXiv:2512.01107.

Optional:

Wu, S., Choi, E., Khatua, A., Wang, Z., He-Yueya, J., Weerasooriya, T., Wei, W., Yang, D., Leskovec, J., and Zou, J. HUMANLM: Simulating Users with State Alignment Beats Response Imitation. https://humanlm.stanford.edu/HumanLM_paper.pdf
Churchland, P. M. (1981). Eliminative materialism and the propositional attitudes. the Journal of Philosophy, 78(2), 67-90. https://www.jstor.org/stable/pdf/2025900.pdf

Salinger.

Posted on March 10, 2026 9:43 AM by Andrew

I’ve been reading some biographies lately–they’re perfect bathroom reading, you can read an episode at a time in someone’s life and over the period of a few weeks get a sense of the whole. Recently I read a biography of J. D. Salinger. As you probably know (if you’ve read to this point in the post and haven’t already clicked away), Salinger lived a sad life: ups and downs in his childhood, traumatic military experiences, a short period of productivity and success as a writer, and then several decades of loneliness.

I’ve always been sad that Salinger wasn’t able to follow up The Catcher in the Rye with anything else of that quality. In my opinion, Catcher deserves is enduring reputation; I think it’s just wonderful.

But after reading this biography, I’ve flipped my take–not on The Catcher in the Rye, which I still think it’s great!–but on my view that it was sad that he wasn’t able to follow it up with work of comparable quality.

My current view is that, Catcher aside, Salinger was an OK writer. Given what he had to work with, it’s amazing that he was able to write that one brilliant book. Rather than being sad he couldn’t write more, I’m happy that he produced what he did.

Let me put it this way. It seems that, as an individual, Salinger was an amazing person, full of talent, the kind of person who, if you met him, might well be the most impressive person you ever meet. But, as a writer, he was just fine, not amazing, except for that one book. Don’t get me wrong, some of his stories are great too, but, overall, after reading his biography, my sense is that Salinger didn’t have that much to say. It’s easy to envision a slightly different world in which he became a successful and beloved writer of TV sitcoms.

The best analogy I can think of is Shirley Jackson–hey, I read her biography too! Jackson wrote The Lottery and some other things. Unlike Salinger, she died young so who knows what would’ve come next, but overall I feel like it makes more sense to appreciate her best work rather than to have expected that much more work of that quality would’ve come.

P.S. In contrast, I remain sad that William Saroyan’s work declined after his wonderful early stories. I don’t think Saroyan’s best work is as good as Salinger’s best work, but I somehow have the impression that, had his life gone slightly differently, Saroyan could have kept writing new stuff at his best level.

“The idea of Israel” . . . more generally, The idea of X, for different values of X

Posted on March 9, 2026 9:03 AM by Andrew

We were talking about herrenvolk democracy in class last week, and I happened to come across a book, The Idea of Israel, published in 2014 by Ilan Pappe. He’s an historian from Israel, now working in England, and his book focuses on the period in the 1990s when it became prevalent in Israeli academia, arts, and the news media to challenge the standard patriotic (“Zionist”) view of the history of Israel/Palestine. Pappe argues that before the 1990s there were almost no prominent anti-establishment voices, then there was a blooming in the 1990s (the “post-Zionist moment”) followed after the failure of the Oslo peace process by a “neo-Zionism” in which dissenting voices have been quieted.

It’s an interesting story and it makes me wonder how it would work in other countries.

U.S. academia has its share of opposition to established narratives, with a tradition going back at least to Charles Beard in the early twentieth century and continuing today, and we see some of this in the news media as well, as least in the soft version such as proffered by TV documentarian Ken Burns. Maybe one reason for this is that the centuries-old political divisions between the North and the (white) South have been accompanied by alternative national myths. So, once you get past 1783 or so, you have two established narratives to choose from, which means that in some ways just about everyone is oppositional. That said, as someone who was born and raised in this country, it’s hard for me to put myself in a state of equipoise where I can read about the American revolution, the First World War, the Second World War, or the Cold War, and not root for our side. I wouldn’t deny that atrocities were committed on all sides of these conflicts, so I’m not saying I’d drink the straight-up 200-proof patriotic history here, but I still have that bias. Then again, I’m not a historian. If I were, and given my temperament, I’d probably just kinda get technical and avoid taking a stand on the big issues.

What about other countries? I have no idea what academia and the news media are like in Egypt, say, but my guess is that they depart from the national party line much less than their counterparts in Israel, let alone in the U.S. What about our neighbors? I’d guess that Canadian academia and culture are similar to the U.S. in broadly supporting the national myths while allowing a fair share of dissent. Mexico is like the U.S. in a different way in that revolution and conflict are part of its national culture: unlike in Israel or Egypt, I doubt there’s a single dominant national myth in Mexico, so I imagine there’s space in Mexican academia and culture for a variety of takes on the country’s history.

There have also been changes over time. I’ve been reading the memoirs of Raymond Aron, a French sociologist who was highly politically engaged for much of the twentieth century, and his stories of the tumult in French academia and news media in the late 1950s during the war in Algeria reminded me of Pappe’s discussion of Israel. There were dissenting voices in France at that time, Aron’s among them, but the dissenters got a lot of angry pushback: they were attacked as traitors, and the opposition was not a comfortable place to be. Perhaps if Charles de Gaulle had been assassinated and a hard-line government had taken over, France in the 1960s might have moved to a neo-patriotic period of cultural repression in the mode of Israel’s neo-Zionism.

After finishing a book I like to triangulate by reading some reviews to get other takes. Directly googling Pappe’s The Idea of Israel was not helpful because all I got were brief positive reviews that didn’t add anything to the book. But more searching yielded some harsh negative reviews of some of Pappe’s earlier books. Some of these were just empty political attacks, but I came across two meaty negative reviews by Benny Morris, an Israeli historian, roughly the same age as Pappe who followed a different political trajectory. Like Pappe, Morris wrote from the late 1980s onward about Israeli war crimes (covering the period before, during, and after the 1948 war), but then Morris moved to what Pappe would call a neo-Zionist position. So the two historians are now in political opposition, even while they agree on many of the historical facts.

In his reviews from 2004 and 2011, Morris pulls no punches, using terms like “complete fabrication,” “absurd,” “can almost be called a deliberate system of error,” “one of the world’s sloppiest historians,” “he will also simply and straightforwardly falsify evidence,” etc.

I can also point you to Pappe’s response to the 2004 review. After reading Morris’s reviews and Pappe’s response, I have to say that I don’t know what to think. I’m no expert on the history of the Middle East and am even less of an expert on the historians of the Middle East. Further googling led me to this Reddit thread which makes a pretty convincing case that Pappe has indeed been sloppy in his historical writing. And, yeah, I know, Reddit, but, still my current thinking is that Morris has a point. I’d have more respect for Pappe if, in his reply to Morris, he’d had some “my bads” to acknowledge errors or misleading passages in his books.

That said, I still find the intellectual-history aspect of Pappe’s book to be interesting. In chapter 3, for example, he supplies this quote from a “history book on 1948 that was used as the main professional text in Israel for many years — The Edge of the Sword”: “Israel’s victory in the war was a miracle performed by divine authority, by a God who has not deserted his people in their hour of need. . . .” This is a book from 1961, so I guess it hasn’t been the “main professional text” for many years, but still.

Setting aside disputes over the facts, which don’t seem like much of a dispute here, actually–Pappe and Morris seem to agree on most of the key points, with the disputes being over various minor details–, this really seems like a debate about legitimacy. In Pappe’s view the pro-Palestinian historical perspective was marginalized in Israel for many years, was briefly taken to be a legitimate position (but hardly the dominant position), and then became delegitimized. Morris seems to be arguing that the pro-Palestinian view indeed should be illegitimate within Israel. I don’t think he’d argue that nobody in Israeli academia, arts, or the media should espouse such views, but he doesn’t think they deserve any respect, in the same way that I wouldn’t think that any respect would be due to an American political scientist who argued that we should bring back the monarchy and become subjects of King Charles III. Or how I think that fringe scientific views such as HIV denialism or belief in ghosts belong on the fringe.

The question of professional balance, or whatever you want to call it, is tied in with what views are considered to be acceptable. For example, in a history of the Second World War, you want to give all sides in historical perspective but the Nazi racial theories would be historicized, not taken as serious arguments. I guess there’s room in academia for a few hardcore Nazis–the government and social media seem to be filling up with them now–but I think it’s also ok to disrespect them. And from the perspective of Morris and other neo-Zionists, there’s not much difference between Pappe and a Nazi, so they don’t have much sympathy for his complaint that he and other post-Zionists have been pushed to the edge of the Israeli conversation.

Even in a technical field such as statistics, I think some leading academic figures have been getting things completely wrong, and I don’t think they’re doing it on purpose, I just think they’re missing the point, that there’s a disconnect between their theory, their methods, and the applied problems we’re all trying to solve.

What I’m getting at is that, in any space of discussion, there’s a question of how much bandwidth to give different views, which perspectives should be taken seriously and which should be dismissed. National history is an interesting example because we’re immersed in it from childhood and it has serious consequences.

From time to time a misguided but fashionable idea will take over an academic field or subfield in a country. Examples are Lysenko’s biology in the Soviet Union, junk social psychology in the U.S. in the 2010-2015 period, Chomskyian linguistics in the U.S. for a few decades, string theory in physics (ok, I’ll leave that one to others to decide how misguided it is), etc. When this happens in science, we see it as an aberration. But when it comes to history, I’m guessing that extreme nationalism is the usual dominant position and that it is rare for countries to give serious consideration to alternative views.

So you can take this post as a general comment about how to think about “The idea of X,” in environments where the range of discussion can shift a lot in different decades according to political, economic, and social conditions.

Authors of the class “Lorrie Moore”

Posted on March 8, 2026 9:24 AM by Andrew

I’ve just been reading the hilarious novel Banal Nightmare, by Halle Butler, and the hilarious short story, Final Boy, by Sam Lipsyte, both of which follow in what might be called the Lorrie Moore tradition of stories in which a witty, disaffected, downwardly mobile, immature, but fundamentally good person has to navigate a world of people who just don’t get it.

When writing about Lipsyte before, I’ve referred to his protagonists as “literary schlubs,” and I traced the trope back to Joseph Heller. Commenters traced the archetype back to Sancho Panza and some characters from 19th century Russian novels. Separately I characterized Lorrie Moore’s stories as “always seem to be centered around a female character who is witty, thoughtful, and refined, and surrounded by really piggy guys.”

The concept I’m thinking about here is related to but not quite the same as “literary schlub” or “witty social victim.” Here I’m focusing not the journey of the lead character through the story, which is structured as a series of episodes in which he or she ineffectively banters with a series of humorless or boorish personal and business associates amid an environment in which we, the readers, are made excruciatingly aware of the unfairness of the world toward the protagonist, while simultaneously receiving the sociopolitical message that our entire society is self-destructive.

In that sense, maybe these books are in the tradition of the George Orwell of Down and Out in Paris and London and Keep the Aspidistra Flying. But different from, say, Jonathan Coe, who is socially critical too but he won’t novels on a single viewpoint character in this way. The closest to that from Coe would be What a Carve Up!, but even that sour-funny book doesn’t quite follow the Butler/Lipsyte/Moore path of a character trying desperately to make sense of the world through wit, in a sort of distant portrayal of the author as Cassandra-like social commenter who can see all but can communicate nothing to the “dense commuters” (to use Auden’s words) surrounding them.

Bayesian inferences and frequentist evaluations

Posted on March 7, 2026 10:03 AM by Andrew

Martin Forster, Marco Novelli, and Charlie Welch recently posted this preprint, which begins:

We use innovations from the frequentist and Bayesian decision-theoretic sequential experimental design literature to study whether, and when, recruitment to a pandemic-disrupted clinical trial should restart. We consider four frequentist and two Bayesian designs, two of which are new, and apply them to data from the UK’s ‘DISC’ trial, a publicly-funded trial whose recruitment was seriously disrupted by the COVID-19 pandemic. The results delivered by all six designs concur with the DISC trial’s results concerning treatment superiority. However, they do so with different levels of information, owing to different recommendations about restarting recruitment. Referencing work on the seven virtues of good statistical practice, we consider how confronting the same experimental data with a range of statistical models could assist policy-makers tasked with managing non-pandemic clinical trials during a future pandemic.

Christian Hennig pointed me to this because it makes reference to our Beyond Subjective and Objective in Statistics paper and our list of virtues:

I’m happy to hear that our framing of statistical practice in terms of these different virtues, as opposed to the unclear (to me) characterizations of methods as “subjective” or “objective,” has been helpful to these researchers.

I did not try to follow the details of the Forster et al. paper, so I will just offer some general thoughts on their endeavor, which is to compare frequentist and Bayesian sequential experiments in their applied context.

Many years ago Don Rubin pointed out that, although Bayesianism and frequentism may represent different philosophies of statistics, they are not directly comparable as methods. Bayesian statistics is a framework for producing inferences given data and assumptions; frequentist statistics is a framework for evaluating inferences given assumptions. The assumptions in the two approaches can be different; the key point is that Bayesian and frequentist ideas can go together. You can use Bayesian methods to create an inference (for example, an estimate or a hypothesis test or a confidence interval or a probabilistic prediction) and then use frequentist methods to evaluate it. Or you can use non-Bayesian methods to create a inference and then use the Bayesian framework to interpret it as an approximate Bayesian inference under some assumptions.

The pairing of Bayesian and frequentist ideas is not necessary. You can set up a model and do Bayesian inference without any further frequentist evaluation (beyond the automatic property of Bayesian inferences that they have the correct frequency properties when averaging over the assumed prior and data distributions). Conversely, you can perform frequentist evaluation of an inference that was not constructed from any Bayesian model; indeed you can sometimes construct purely frequentist inferences derived from some non-Bayesian principle such as minimax loss.

But, if you are considering various inferences, Bayesian or otherwise, it makes sense to look at their frequency properties. This is a point that Rubin made in his classic paper from 1984.

This is not automatic. Just as Bayesian inference is not a single thing but rather a framework (that is, the inference depends on your prior and data model, not on the observed data alone); similarly, frequentist evaluation involves choices, both in what frequency properties to look at (for example, unbiasedness is often taken as a desirable feature in estimation but does not make sense for probabilistic prediction, a point we make in chapter 4 of Bayesian Data Analysis; see the example on page 94 of BDA3) and in what assumptions to be made about the underlying system and the data-collection process. Bayesian methods are correct when averaging over the joint distribution of parameters and data, and frequency evaluation involves averaging as well: that’s why I say that Bayesians are frequentists.

Just as I don’t think all Bayesian inferences are good (you can have a model that makes no sense, or is inappropriate for the problem under study, or has mathematical artifacts that can be tricky to discover, as in section 3 of this paper), I also don’t think that all frequentist evaluations are appropriate. I’ve already mentioned the problem with applying the concept of unbiased estimation to prediction problems; also I’m on record as generally opposing the so-called Fisher exact test (see section 3.3 here) on the grounds that it corresponds to a data distribution–a generative model for data given parameters–that almost never applies in real life. I similarly don’t like classical multiple comparisons methods, because they are designed to guard against the generally irrelevant condition that all true effects are exactly zero.

So, yeah, doing good Bayesian inference is not always easy–you need to construct that generative model of parameters and data–and doing good frequentist evaluation is not always easy either: you need to think about what’s a reasonable model to use for the data distribution to average over, and come up with a range of plausible parameter values. But it can be done, and, indeed, if you want to compare methods, it must be done. Frequentist evaluation is the only game in town. Although it can be done approximately or implicitly, for example using cross validation or external validation of predictions without ever formally setting up a model to be averaged over.

P.S. The authors respond to the comment thread here.

More on making a killing from prediction markets

Posted on March 6, 2026 9:35 AM by Andrew

Yesterday we discussed the disturbing phenomenon of people betting publicly on people being killed. I pointed this to David Rothschild, an economist at Microsoft Research and occasional collaborator of mine. He wrote:

It’s not actually that hard to design prediction-market contracts that are both informative and that don’t incentivize bad behavior–either inside the exchange (insider trading, price manipulation, settlement manipulation) or outside it (changing real-world actions to “win” the contract: throwing the point spread, or, more extremely, creating incentives around geopolitical events).

The challenge is that doing this well takes expertise, and the contracts that generate the most trading volume are often the ones that are either uninformative (like sports outcomes or what Trump will say in a speech) or create problematic incentives. In other words: the features that maximize activity and revenue can be in tension with the features that maximize information quality and safety.

And that may be the deeper issue: “informative + limited-bad-behavior” may not be compatible with $100B+ valuations and the scale dynamics those valuations imply.

What’s funny is that years ago many of us studied how prediction markets could be accurate with small, independent trading groups and play money (like the Hollywood Stock Exchange). Now the question has flipped: can prediction markets be safe and sustainable when you have large crowds and billions of dollars flowing through the exchange?

More directly on the “death market” issue: Glenn Weyl wrote a piece prior to Brexit (I wish I could find the link) arguing that Brexit contracts on UK exchanges were likely undervalued, because a Brexit outcome would mechanically trigger a sharp depreciation of the Pound. Traders pricing the contract had to implicitly hedge currency exposure, pushing prices away from true beliefs. A similar logic applies here. Any contract that carries a meaningful risk of cancellation, voiding, or nonpayment will, by definition, be inefficiently priced. Traders must discount the payoff not just by the event probability, but by the conditional risk that the contract does not pay as written: introducing endogenous wedges that have nothing to do with beliefs about the underlying event. This is not a bug of trader irrationality; it is a structural feature of contracts whose settlement is contingent on platform, regulatory, or reputational decisions.

I can believe that, empirically, markets with play money could work. But in theory isn’t there a problem that, to the extent that the play markets are trusted and can have impact in the outside world, there will be an incentive to manipulate them?

It seems to me that there is some sort of uncertainty-principle thing, where if the stakes are high there is a motivation to “throw the game” or, more generally, influence policy, but if the stakes are low there is a motivation to manipulate the market.

To put it another way, the ideal for a prediction market is for it to have high stakes (so that it’s costly to try to manipulate the price) with bettors being people with no influence on the outcome. But such a market will be a ripe target for people who can influence the outcome and for insider trading. An alternative is a low-stakes market populated by experts (Rothschild’s “play money” example), but then there is the concern that some of these experts will bet to manipulate the market to serve their own policy preferences, which was an issue for that proposed terrorism prediction market that was going to be run by a terrorist.

This does not mean that I think prediction betting markets are a bad thing–as I wrote in the earlier comment thread, they’re a method of aggregating information–; it just seems that they have fundamental limitations but are often idealized. Economists and journalists such as David Rothschild, Rajiv Sethi, Nick Whitaker, and J. Zachary Mazlish. play a useful role, as both of them are interested in prediction markets and think they have value, and they’re also realistic about the challenges of setting up and understanding these markets.

Here’s another example.

During the months leading up to the NYC mayoral election, there was lots of discussion about prediction markets. We discussed some of the decision analysis here, also there was this discussion where Wall Street guy Bill Ackman straight-up suggested the possibly legal approach of using prediction markets to bribe Eric Adams to drop out of the mayoral race (which Adams eventually did, not that this helped Ackman in any way). At the time, I wrote:

There are lots of ways that Adams could corruptly benefit from strategically dropping out, and indeed these are methods that could be both easier and more lucrative than trying to manipulate betting markets. Indeed, if the government doesn’t want to offer Adams some position for which he is unqualified, Bill Ackman himself could just hire Adams for some no-show job in his organization at a salary of a million dollars a year, no?

So, given everything that’s been talked about in the mayoral election so far, it doesn’t seem that prediction markets represent much of a change to the moral, financial, and political calculations of corrupt maneuvering in a political race.

Meanwhile, Palko points out that the courts have gotten very non-judgmental about corruption, which is related to an ultra-free-market ideological position in which any activity done by rich people is ok and indeed desirable in that it increases market efficiency.

Sethi points out that you wouldn’t be able to identify the traders on Polymarket even if insider trading were illegal, all you have is a wallet ID. And there seems to be a lot of it going on, including on the Comey indictment, military strikes and even (possibly) a celebrity engagement.

My summary: when it comes to politics, I think the Polymarket thing is just an amusing distraction. It’s not so much that betting markets represent a threat to democratic governance–direct payoffs from rich guys to politicians seem like much more of a threat, and as Mark notes, this has become basically legalized, at least for now and for the right people–but rather that threats to democratic governance are also threats to prediction markets.

A big appeal of open-ended prediction markets is that they’re an opportunity for people to make money unethically (whether through insider trading or by taking advantage of suckers). Another big appeal is ideological, for people who think markets are Jesus.

P.S. I also recommend this 404 Media podcast on the topic.

Making a killing from prediction markets

Posted on March 5, 2026 10:37 AM by Andrew

Paul Alper points us to this news story from the Washington Post, “Bettors wagered $54 million on Khamenei’s death. Now they’re not getting paid,” which reports:

When he learned Saturday about the killing of Iran’s Ayatollah Ali Khamenei, the Israeli-American business executive in New York was excited to cash in.

On the prediction-market site Kalshi, the executive – who spoke on the condition of anonymity due to concern over what his friends would think – had placed two bets, totaling $3,460, that Khamenei would be “out as Supreme Leader” by March or April 1. His Kalshi app placed green check marks next to his bets, indicating he’d won payouts worth more than $63,000.

Minutes later, however, Kalshi froze the $54 million trade for everyone who bet on that scenario, saying the site doesn’t allow transactions “directly tied to death.” The change triggered an online uproar, as Kalshi users flooded social media to argue the site had unfairly robbed them of winning bets. . . .

Kalshi heavily promoted the trade to bettors on its homepage and app and in push notifications before Khamenei’s death was publicized. Kalshi also tweeted the morning of the strike that the odds “Khamenei is out as Supreme Leader have surged to 68%,” along with a disclaimer that Kalshi didn’t broker trades that “settle on death.” . . .

Polymarket said in August that the president’s eldest son, Donald Trump Jr., had joined its advisory board, and a handful of recent bets on the administration’s moves have sparked public accusations of insider trading.

The analytics firm Bubblemaps said it found six “suspected insiders” on Polymarket that had made $1.2 million by betting that the U.S. would hit Iran by Feb. 28, the date that Operation Epic Fury began. All of the accounts were made last month and bet exclusively on Iran-strike timing; some of the bets were made within hours of the first explosions in Tehran. One account bet $60,000 and won $560,000. . . .

On the first morning of the assault, Polymarket posted a meme image of a man with five screens laying out bets about Khamenei’s ouster and the caption, “Can’t right now babe, I’m monitoring the situation.”

OK, this is just evil. I was going to say, “Even setting aside the corruption angle . . .”–but why should we set aside the corruption angle? As we’ve seen from the news over the years, you don’t need prediction markets to have massive corruption, but this seems like just one more opportunity for these people to loot the public treasury. Indeed, as we’ve discussed, this sort of thing is literally meritocracy (in contrast to a merit-based employment system in the civil service).

So, yeah, that’s bad. But even setting aside the corruption angle, the idea of betting on someone’s death is really disturbing to me–and not just to me, of course, hence the controversy evidenced by the news article.

Now, I recognize that my perspective is culture-bound. Back in the day, life was cheap, humor was coarse, and gambling has always been popular. Ancient Romans bet on gladiators, and that was a life-and-death sport. And, even nowadays, you’re allowed to bet on your own life. Indeed, a few years ago I made such a bet myself, putting money up front with a big payoff if I die before the age of 65.

Where to draw the line in this particular family of trolley problems? Life insurance is ok, as long as no fraud is involved–and, as I’ve learned from various movies of the noir era, you should choose your beneficiaries wisely. Buying life insurance for someone else, that’s not so cool.

What about betting on which of the people in this list die first? That’s in poor taste but it doesn’t seem evil to me (assuming you don’t have any personal connection to these people); it’s an example of an actuarial calculation, and actuarial calculations seem just fine to me. Indeed, it’s even fine with me to gamify this, as, after all, insurance companies make a lot of money from this sort of thing anyway.

Even with actuarial work, there’s the potential for market manipulation, as various government policies can affect death rates (hi, Secretary Kennedy!), but this is nothing compared to the corruption possibilities for things like government contracts and drug approvals; I’m not so concerned about it.

I could imagine some situations where it would not seem evil to bet on individual people dying, people you actually know. I’m thinking of settings where life is cheap, and you’re personally at risk. For example if you’re part of a platoon of soldiers on some suicide mission, I could imagine a grim, gallows-humor sort of bet on how many of you will come back alive. Presumably the cost-benefit here is such that there’d be no motivation for market manipulation. On the other hand, it would seem evil to me for two officers to bet on the lives of the soldiers they’re sending on the mission, and it also seems absolutely wrong for two doctors, say, to bet on the survival of a patient. Even if this does not affect the doctors’ performance and they’re just doing it to let off steam, no, I don’t think this works in our society. Grim jokes, sure–I’ve read enough hospital-themed books and seen enough hospital-themed TV shows to get the impression that doctors, like cops, will joke about life and death in ways that might seem to outsiders such as me to be in poor taste–but betting, no. That’s real screwed up Dr. Oz-level shit.

There’s also a difference between betting that someone dies during the next year–that can happen if you’re 112 years old!–and betting that someone will be murdered or assassinated. The real issue isn’t bets “directly tied to death,” but rather bets directly tied to killing.

What about betting on economic statistics like the inflation rate? No life and death here, and it seems ok, but I do think it should be regulated, in the same way that bank loans are regulated and betting on the stock market is regulated. There’s nothing stopping people from doing such bets on the side, but when real money is involved . . . you can’t eliminate all possibilities for corruption, but you at least want to make it difficult, balance the expected costs and benefits so that the incentives for market manipulation are reduced.

Similarly with betting on political events such as elections and, yes, wars. People are already betting on wars indirectly, through “put options” on oil futures, or whatever it’s called. Setting aside any issues of life and death, the corruption issue becomes more and more of a potential problem to the extent that the bets become tied to particular actions by individuals. Just for example, you might want to take an over/under bet on this blog’s total web traffic in 2026 (actually, I have no idea what our traffic is; I purposely didn’t set up such a tracking system because that way lies madness), but you wouldn’t want to bet on something directly manipulable such as whether I post something in the Literature category on 28 March.

Besides the distastefulness of betting on a particular person being killed, there’s also a problematic history of these betting markets. As we’ve discussed before, back in the early 2000s the government tried to set up a terrorism prediction market that was run by an actual terrorist. The problem in that case was not financial but rather the fox-guarding-the-henhouse issue. Talk about moral hazards. This is not a joke; it’s people in power who seem all too willing to consider other people’s lives as expendable. Which, yeah, that’s how things are in a war. But often the decision to go to war is itself a choice.

P.S. More on this from my Columbia economics colleague Rajiv Sethi.

P.P.S. More here.

Missing forecasts

Posted on March 4, 2026 7:28 PM by Lizzie

This post is by Lizzie.

Greetings from sunny Vancouver! I saw a magnolia in bloom today, which feels super early to me. But less early in that cherry and other plum trees have been blooming around the city. We have not had the coldest winter and had a recent warm-ish spell with lots of sun … so maybe that’s the trigger? The usual ‘two bucket’ model for spring flowers is that plants need to fill a bucket of winter cool (cool, but not cold, what this bucket is doing is covering some sort of latent variable related to ‘chilling,’ a possibly mystical biological concept*) and then a bucket of spring warmth (this makes more sense — warmth leads to cell growth and division). A little sun would likely help that second bucket.

Figuring this out of course is the goal of the International Cherry Blossom Prediction competition, which Jonathan Auerbach, David Kepplinger and I run.

Predictions just came in** and they tell me to hold my breath a while on Vancouver but that blossoms are coming this month probably. And I should wait a month for the east coast too — I hear it has been a cold winter there. We’re not far off the Washington Post’s predictions (which just came out) and of course we’re hoping to beat them.

We’re not sure how we compare to the National Park Service’s forecasts. They have had one each of the four years before when we ran the competition but seem quiet this year. I could imagine what may have happened but if anyone actually knows, let me know in the comments.

BTW, the bloom video is of Hibiscus, not cherries (if you want to visualize how far apart they are on the tree of life, go to https://www.onezoom.org/ and then type in ‘Hibiscus’ in the search, pick the first one, then type ‘Prunus’ and pick your favorite from the list, it will zoom you over). And big thanks to Tim Savas for the amazing video.

* I am hopeful we’re getting closer on ‘chilling’ — the best idea so far is that it’s capturing some enzyme or such that breaks down the callose (which I call a sugar, but someone told me is better defined as a polysaccharide) that blocks the plasmodesmata (the little wholes in cells that let them communicate with each other). A new study has new info and suggests this whole thing could connect to bet-hedging in plants … I am dubious about that last part.
**If you have comments on the figures, see also here.

Ethics corner: “As a statistical consultant, if you’re a co-author on a substantive paper, is it your duty to ensure that all the possible statistical concerns you could imagine are addressed?”

Posted on March 4, 2026 9:31 AM by Andrew

Michael Truong writes:

I had this question regarding statistical consulting that I thought might also make some content for your blog:

As a statistical consultant, if you’re a co-author on a substantive paper, is it your duty to ensure that all the possible statistical concerns you could imagine are addressed? Or is it kosher to only solve the specific statistical problem that you were brought on to solve? What could be the consequence of having your name associated with a paper that has some poor statistics, even if that poor statistic has nothing to do with what you were hired to do? What happens if addressing your statistical concern implies a lot of statistical thinking (work) that you aren’t prepared to personally handle?

My reply:

I think your first moral obligation is to be honest: tell the client all the relevant analyses you did, and don’t say you did anything that you didn’t. This rule also works for writing your report: just clearly write down everything you did, all your steps.

The next question is, what if you think the client should’ve done something different? It could be that you think they did something wrong (statistically inefficient, erroneous, incompetent, or even unethical) or that you recommended an idea that they refused to adopt. When this happens, sometimes you just have to suck it up. Even if you think the client’s work is professionally unethical–for example, a Wansink-style data dredging exercise–, as a consultant you don’t necessarily have an option to stop this from happening, nor will any good whistleblower programs typically exist.

Coauthoring is another story. You should always have the opportunity to remove your name from a paper that is submitted or being revised for publication. Unless, I guess, your consulting contract specifically requires you to be a coauthor no matter what–but I guess I’d recommend not signing such a contract!

Finally, to respond to the last sentence of Truong’s question: if you feel that the best step will require more work than you are currently able or prepared to do, then that’s fine, just directly tell the client directly, who can then decide whether to follow your advice and hire someone to do that additional work, or to just stop right there. It’s the client’s call, and there’s no reason for you not to be honest about what you would recommend for the client, going forward.

The other reason I recommend honesty in such situations, beyond straight-up ethical reasons, is that business relationships, like personal relationships, require some level of compatibility. Be open about your thoughts, and if the client doesn’t value this, better to learn this sooner.

In all the above advice I’m assuming you have some employment flexibility. If consulting is your entire job and this is one of some small number of clients and you need this gig to put food on the table, then all bets are off.

Survey Statistics: sampling-weighted loss

Posted on March 3, 2026 4:00 PM by shira

We’ve mostly focused on a population mean E(Y) as our quantity of interest. We saw how methods extend to estimating a subgroup mean E(Y | V=1), e.g. voters.

What about estimating a general conditional mean E(Y | X) ? We talked a lot (4 posts) about calibrating this to a known population mean E(Y), e.g. via the “logit shift”. But first we start with an estimate of E(Y | X) from survey data.

Lumley 2010 Section 5.2 says:

The polar bear has been going thru the pile of papers he was sitting on last week and found this:

Replace R (whether you respond to a survey) with T (whether you are treated) and you can see that my drawing is heavily inspired by Johansson et al. (2022) Figure 3:

We’ve talked about connections between survey random sampling and randomized experiments. There are also connections between nonprobability surveys and observational studies. We will explore more analogies between survey statistics and causal inference. Favorite references ?

“The inner workings of a scam”

Posted on March 3, 2026 9:17 AM by Andrew

This story from pharmacologist Csaba Szabo is hilarious/horrible. He goes through a long email exchange with a fake-academic-journal scammer, gradually extracting various aspects of their evil business plan.

I get these scam messages all the time and either ignore them or mock them on the blog (as in this notorious example which came from Wolfram Research). Sometimes I try something cute like, they tell me about some service they have to turn my work into a fancy brochure and website for the cost of a mere $3000 or whatever, and I respond: Thanks, but $3000 isn’t enough, if you send me $10,000 I’ll consider. So I admire Szabo to have gone to all this effort to expose this for us.

Olympic memories

Posted on March 2, 2026 9:19 AM by Andrew

From 2024:

Here’s my suggestion for next time: After all the events are over and the medals have been given out, do a series of events with the medal winner of sport A, competing against the medal winner of sport B, doing sport C. With A, B, C drawn out of a hat. So we could see a ping-pong champion vs. a wrestler in a diving event. Etc. This would be so cool!

From 2023:

After we got back, people would ask what we had seen at the Olympics. I would say “We saw Usain Bolt run the 200m, we saw the women’s 4x100m relay and the men’s 4×400, we saw the last events of the decathlon…lots of great stuff. But my favorite was the men’s 800m.” . . .

From 2021:

Tokyo Track revisited: no, I don’t think the track surface is “1-2% faster”

From 2013:

How fast do we slow down? . . . For each doubling of distance, the world record time is multiplied by about 2.15. . . . for sprints of 200 meters to 1,000 meters, a doubling of distance corresponds to an increase of a factor of 2.3 in world record running times; for longer distances from 1,000 meters to the marathon, a doubling of distance increases the time by a factor of 2.1. . . . similar patterns for men and women, and for swimming as well as running.

From 2012:

I suppose it’s too late to add Turing’s run-around-the-house-chess to the 2012 London Olympics?

From 2010:

The Whiter Olympics. . . . And they’re not talkin bout the snow, either. . . .

Did you know that Puerto Rico had a Winter Olympics team? One year it featured my cousin Bill, who finished last in the slalom. I’m pretty sure he wasn’t born in Puerto Rico (despite what it says on one website), but I guess he’s probably been there on vacation on occasion. And I wouldn’t be surprised if he speaks Spanish–he does live in L.A., after all. And, of course, it takes some skill to finish last in the slalom. I’d probably fall off the chairlift and never even get to the starting line.

From 2006:

The overseers of international figure skating scoring instituted a new system in 2004, designed to reduce the chances of vote fixing or undue bias after the scandal during the Winter Olympics in Salt Lake City in 2002. Under the old rules eight known national judges scored a program up to six points with the highest and lowest scores dropped. Under the new rules, 12 anonymous judges score a program on a 10-point scale. A computer then randomly selects nine of the 12 judges to contribute to the final score. The highest and lowest individual scores in each of the five judging categories are then dropped and the remaining scores averaged and totaled to produce the final result. . . .

From 1945:

If you wanted to add to the vast fund of ill-will existing in the world at this moment, you could hardly do it better than by a series of football matches between Jews and Arabs, Germans and Czechs, Indians and British, Russians and Poles, and Italians and Jugoslavs, each match to be watched by a mixed audience of 100,000 spectators. I do not, of course, suggest that sport is one of the main causes of international rivalry; big-scale sport is itself, I think, merely another effect of the causes that have produced nationalism. Still, you do make things worse by sending forth a team of eleven men, labelled as national champions, to do battle against some rival team, and allowing it to be felt on all sides that whichever nation is defeated will “lose face”.

Meanwhile, tug-of-warriors haven’t been allowed into the five-ringed halls since 1920.

Remembering David MacKay

Posted on March 1, 2026 9:44 AM by Andrew

Pilgrim Beart informs us that is Cambridge Philosophical Society is organizing a meeting on Energy and Information to mark the 10th anniversary of David MacKay’s untimely death. Beart asked me to share any tribute to MacKay, and I sent him this:

In his classic 2003 book on information theory, David wrote that, “in many problems, we really only need about twelve independent samples.” He explains: “Now, how accurately would a manager like to know [a parameter] Φ? I would suggest there is little point in knowing Φ to a precision finer than about σ/3. After all, the true cost is likely to differ by ±σ from Φ. If we obtain R = 12 independent samples from P(x), we can estimate Φ to a precision of σ/√12 – which is smaller than σ/3. So twelve samples suffice.”

This led me to wonder . . . . . . where did the “12” come from in MacKay’s passage? When I first saw the number there, I assumed it would have something to do with the variance of the uniform distribution. But that doesn’t seem to come up at all. Accepting MacKay’s stipulation that σ/3 would be enough precision in this sort of example, shouldn’t he have said that 9 random draws would suffice? I could see him rounding that up to 10. Or even 16, if he wanted to get all base-2 on us. But why 12? I didn’t get this at all.

So I asked David. Here was his reply: “You said that can imagine rounding up 9 to 10 – which would be elegant if we worked in base 10. But in the UK we haven’t switched to base 10 yet, we still work in dozens and grosses. . . . Probably in an earlier draft of the book in 2001 I said ‘a dozen’, rather than ’12’. Then some feedbacker may have written and said ‘I don’t know what a dozen is’; so then I sacrificed elegant language and replaced “dozen” by ’12’, which leads to your mystification.”

This reminds me that, when I was a kid, my dad told me that in Britain there was a Duodecimal Society, devoted to promoting base 12. This was told to me in a “There will always be an England” sort of thing. In writing this note, I was curious so I looked it up on the web, and it turns out that there is such a society, but it was first established in the United States. So there! I guess David was more American than he realized.

I thought that would be better than ranting about why I agree with Radford Neal and disagree with David MacKay about Occam’s razor (although David claimed I actually agreed with him). I guess I can send them that story in the next commemoration, a decade from now.

It’s actually a good story in that I thought we disagreed, but David disagreed with me on whether there was a disagreement.

David Mackay was a smart, generous, and committed person. It’s good to remember that such people exist: not every important person science is like this guy or these guys or these guys or these guys. The greedy people who did all those things often seem to feel like their behavior is ok because everybody does it. David Mackay is a reminder that, no, better things are possible.

A link to the conference is here. There’s also a form where you can share your memories of him.

Herrenvolk democracy and objections to ethnic political representation

Posted on February 28, 2026 9:17 AM by Andrew

I go onto Twitter and Bluesky every morning to post the link to the new day’s blog post. When I go on Bluesky, I know what to expect: it shows me a list of my recent posts. Twitter does something different: it gives me some general-interest feed, some mix of home improvement videos, celebrities, politics, and whatever has recently liked by Elon Musk. The above post came up this morning.

Usually I find such posts annoying, but this one was convenient because it overlaps with some of what we’re covering next week in our course, Rationalizing the World: The Hopes and Disappointments of American Social Science, 1900 to the Present:

Week 6: Democracy and totalitarianism

The new challenge of the 1940s was the cold war: fascism was defeated, but communism remained and was stronger than ever. The ideological struggles of the twentieth century led scholars to study different forms of modern government and how they have developed in the United States and elsewhere. We consider the connections between various forms of democratic and authoritarian government and their connections to domestic politics and colonialism.

In particular, we have several readings related to “herrenvolk democracy,” which is the term used for countries such as apartheid South Africa and the U.S. south that had democratic governance but with the vote restricted to members of the ruling ethnic group. We compare this practice to universal democracies, completely non-democratic systems, and colonial systems such as the former French Algeria, the territories controlled by Israel, American Indians in U.S. territory, and various past restrictions on suffrage based on property ownership and sex.

I can’t be sure, but I think that in the above post, Musk is expressing support for a sort of politics of ethnic separation, where Pakistan is populated by ethnically Pakistani people with Pakistani accents and wearing Pakistani clothing and Scotland is populated by ethnically Scottish people with Scottish accents and wearing Scottish clothing.

It’s easy to poke holes in this sort of thing–for one thing, “Pakistani” is itself a relatively new nationality; for another, there are ethnic divisions even among national groups; for another, Musk himself lives in the U.S. but he’s from South Africa–I guess this is OK by him because he now dresses like an American, and his South African accent is not too thick, but then of course it raises the question of how his nationality principle would work in a multiethnic country such as the United States or South Africa which have many languages, accents, and clothing styes. At this point perhaps his reply would be that the U.S. and South Africa would be better off if they were not multiethnic, but it’s not clear what could be done about that. It’s not like you could split the countries into separate, homogenous bits: all sorts of people live in all these places. Beyond all this is the history of colonization, in particular including Musk’s example of Pakistan and Scotland: as the saying goes, “We are here because you were there.”

OK, fine. A political slogan doesn’t have to be logically coherent: it just needs an emotional appeal coupled with some connection to a current political dispute–in this case, Musk wants people to vote for anti-immigrant parties in the U.S. and the U.K. I think he’s holding Pakistan as a sort of ideal, in the sense there are not a lot of people of European descent living there, so they don’t currently have to worry about ethnic outsiders holding political office.

But let’s step back and think about this in terms of political theory. Herrenvolk democracy isn’t on the table in the U.S. or even in South Africa. So what would Musk’s ideal be here, beyond his goal of not letting more people enter the country who he would consider suitable in their politics, ethnicity, language, and clothing? Maybe some system of separate ethnic voting, where different groups vote for their own political leaders, and then these leaders of the different group negotiate in some sort of cabinet? I think they had this system in Lebanon for many years, also Belgium is divided up in this way.

The point here is that we can think of herrenvolk democracy not just as a historical throwback (or, in the case of Israel, as a sort of living fossil) but rather as a response to political conditions, an available solution to a problem perceived by some powerful people.

In this course, I’m trying to help students see social and political events, and social science theories, in their historical contexts, and this is a good example. To someone in Musk’s position, straight-up democracy is unacceptable, but he recognizes that any system needs some form of legitimacy, and that’s leading him in this direction, of centering legitimacy in ethnicity. His South African background is part of the story, but given how many countries have herrenvolk traditions, you could just as well say the same thing if his background were English, or French, or American, or all sort of other nationalities.

Again, I’m not saying that Musk is wrong, or that he’s right, to be upset that Scotland has enough ethnic diversity to support the election of politicians who don’t dress the way he likes. People are free to be upset about whatever they want. It’s more that this is an example of the sort of elite panic that can motivate social science theory as well as policy.

Hey! What the Froot was up with this Harvard website?

Posted on February 27, 2026 9:02 AM by Andrew

This is probably the least important topic in the history of our blog.

I have this old post, Teaching materials now available for Llaudet and Imai’s Data Analysis for Social Science!, which recently received the following comment from Shriram Krishnamurthi:

For whatever strange reason, the Website link (for “tons of materials”) now redirects to an unrelated(?) professor’s vanity site. The link one wants is here:
https://ellaudet.github.io/dss_instructor_resources/

I checked and Shriram is right, so I fixed the link.

But here’s the weird part.

The old link (now removed from the earlier post) was: https://scholar.harvard.edu/ellaudet/dss-instructor-materials

I get that Llaudet is no longer at Harvard and they might want to keep their website clean by removing links to former affiliates. So it makes sense that the link “https://scholar.harvard.edu/ellaudet/dss-instructor-materials” would no longer work.

But here’s the weird thing. Go to that url and it bumps you over to a website called “https://k-froot.com,” which looks like this:

It’s the homepage of a retired Harvard professor of business administration. I tried plain old “https://scholar.harvard.edu/ellaudet” and this also takes us to k-froot.com.

And then I tried “https://scholar.harvard.edu/abc” and, you guessed it . . . it takes us to k-froot.

Did Froot make some sort of arrangement with his employer so that all non-working links at https://scholar.harvard.edu go to his webpage? This would seem a bit bizarre, as it’s not like he’s getting anything from these links (in case you’re wondering, if you click through to his C.V. you’ll learn various random things such as that in 2016 he received the “Crowell Second Paper Prize, PanAgora Asset Management”).

Or maybe it’s a glitch on the Harvard website, that these nonworking links all get sent to k-froot.com because some accidental bit of code ended up in their html? I have no idea. Nonworking links at the regular Harvard site (e.g., https://www.harvard.edu/abc) just go to your standard 404 page (not the one linked here, unfortunately). Does anyone have any idea?

P.S. I just checked the links and that Froot thing is no longer happening. Maybe it was some temporary glitch that they fixed. Weird. Anyway, I changed “is” to “was” in the title of this post.