“If you decide to write about this paper in your blog, please don’t mention my name. It is important to me to remain anonymous. . . . exposure might harm me or my students.”

So, this came in the email one day:

I am a social psychologist, and I was asked to review a PNAS paper . . . I don’t consider myself an expert in statistics, but I could easily detect three types of issues, which according to your blog are not uncommon in this journal:

– Overstatement of findings
– The journal’s willingness to accept very weak findings (e.g., marginal effects, deviations from preregistration on the key analysis)
– Dishonest reporting: presenting exploratory analysis as if it was preregistered, presenting preregistered hypotheses that were not confirmed as if they were exploratory, deviation from preregistration that is not acknowledged (e.g., not using an important preregistered covariate that should have been used). . . .

In my understanding, the paper creates the impression that a certain intervention . . . is effective, while in fact the evidence suggests otherwise. Personally, I spent a lot of time, effort, and money of funding agencies on an intervention that didn’t work, and I think that it’s important that as a field we move forward and abandon ideas that are wrong. . . .

My correspondent continued with details of various analyses in the paper that were not consistent with the preregistration and then added:

If you decide to write about this paper in your blog, please don’t mention my name. It is important to me to remain anonymous. . . . exposure might harm me or my students.

My reply: Yes, without looking at all the details, the general impression you give reminds me of other examples of this sort, unfortunately.

I’m not linking to the article here, partly to preserve anonymity but mostly because the point here is how commonplace this all is.

“Death and Taxes: Taking Causal Inference into the Real World”

Nandita Mitra is giving a webinar this Tuesday at noon with the above title and the following abstract:

Statisticians develop methods with the hope that they will ultimately inform decisions and improve the world around us. But taking causal inference into the real world requires more than choosing the right estimator: it requires identifying the question that matters, engaging deeply with its context, and communicating statistical reasoning to very different audiences.

In this talk, I will draw on two experiences in which causal inference took me well beyond traditional statistical applications. I will first discuss my experience serving as an expert witness in a death penalty case involving racial disparities in jury selection, where ideas including confounding, mediation, and individual- versus population-level effects had to be translated into compelling and understandable arguments in a court of law. I will then describe my involvement in evaluating Philadelphia’s excise tax on sweetened beverages, where cross-border purchasing led us to confront interference and develop methods to better inform policy decisions.

Through these examples, I will reflect on how statisticians can seek opportunities to bring their expertise to consequential real-world problems, and on the importance of collaboration, communication, and stepping outside our usual academic boundaries. Most importantly, you don’t have to choose between being a methodological statistician and having an impact outside statistics.

This looks interesting. Click on the link above to register for the talk and get the zoom link.

“I write to seek your help and advice regarding a published article that my group and I believe is invalid and should be corrected or retracted.”

David Allison writes:

I write to seek your help and advice regarding a published article that my group and I believe is invalid and should be corrected or retracted. Despite our concerns, the editor seems to be unwilling to accept responsibility and to be responding to us in a way that we perceive as unethical.

You may wish to see a full account on this matter including the Letter to the Editor (LTE) that was rejected by the journal [Di Molfetta et al LTE (short version) Submit.pdf (attached)], our email exchange with the editor following its rejection [Diabetes Research and Clinical Practice Editor Email Exchange.pdf (attached)], and our email exchange with the authors and editor prior to submitting the LTE [Di Molfetta Email Exchange.pdf attached)], but a succinct summary of the events are provided below for brevity:

07/30/2023: We identified a paper as having errors related to not properly analyzing cluster randomized trial data.
08/08/2023: We contacted the original study authors and requested the deidentified data to reproduce the published results.
08/21/2023: The authors requested clarification for the reason for the data request.
08/21/2023: We provided clarification for the data request.
09/22/2023: We prompted the authors to reply to our prior communication.
09/30/2023: We included the editor in the email exchange requesting that they uphold the journal’s data sharing policy.
10/14/2023: We prompted the editor to reply to our prior communication.
10/14/2023: The editor replied and asked for clarification for the reason for the data request.
11/03/2023: We clarified that we requested the data to reproduce the analyses and re-analyze the data using the correct statistical methods for the study design. We also provided a summary of the errors we identified.
11/23/23: The authors replied with their rebuttal to sharing the data, rationalization for their errors, and rationale for not correcting them.
04/29/24: We submitted a letter to the editor to the original journal.
05/01/24: The manuscript was rejected on the basis that our “paper cannot be assigned sufficient priority to allow publication.”
05/08/24: We wrote to the editor directly to reconsider publication of our letter.
05/09/24: The editor responded that the journal does not wish to be engaged in a legal issue and that the matter will not be entertained by the journal any longer.
05/10/24: We responded to the editor to clarify that we simply want the journal to publish our letter and had not proposed any legal action.

The editor has not responded to our last communication as of 05/21/24.

We believe that the most recent response from the editor is patently unethical and incommensurate with the publishing standards of Elsevier, guidelines of COPE, and generally accepted norms in the field of science.

We have two requests:

1. Do you have any advice for us on how we might best pursue this issue further.

2. Would you be willing to blog about this item? If so, you might compare and contrast it with another recent case we pursued [BMC Public Health LTE Example.pdf (attached)] which was an exemplar situation where the journal was responsive to the ethical issues we raised.

I don’t have any advice on what they should do! The flood of bad research is just so overwhelming. But I can blog it. So I did.

David Allison often works in nutrition science, an area where our government is spreading misinformation that is then sane-washed in the news media. And, along with all that, there’s so much uncertainty and variation that it’s hard to know what to think or do.

Why do we like wind chimes?

Dmitri Tymoczko writes:

You know your question, “why do we like to listen to music over and over again?”

I think the question reveals that our relation to music is sensory and pleasure-based rather than cognitive and information-based. Music (like food and alcohol and sex) gives us pleasure, which is why we do it over and over.

Anyway, I am writing because there is a mirror version of the question: “why do we like wind chimes?” Wind chimes are essentially random notes, but people like them. We don’t really like random words, because our relation to words is information based.

I wanted to title this post, “Why do we like to ask, ‘Why do we like to listen to music over and over again,’ over and over again?”, but I decided that was too cute so I switched to the more SEO-friendly version above. Everybody likes wind chimes, right?

To answer Dmitri’s question, my personal theory of “Why do we like to listen to music over and over again?” is that there’s a big difference between listening to music actively (or playing music), and listening passively (which is how I usually do it).
Continue reading →

Seeking to interview social scientists experimenting with LLM simulations of opinions or behavior

This is Jessica. As part of our interest in how social scientists are using LLM simulations of human behavior, some collaborators and I are seeking interview participants. We are thinking about tools to support rigorous design and evaluation of such studies, and would like to hear examples of goals and challenges that researchers who are experimenting with LLM simulations are facing.

Specifically, we’re looking for researchers in academia or industry who are currently using LLM simulations in their work and could bring a simulation dataset (and optionally, human ground truth data) that they are working with to the interview. We are interested in use cases like piloting surveys or experiments with LLMs, replacing or supplementing human participants in surveys or experiments with LLMs, or using LLM simulations for post-study analysis or theorizing. The interviews should take no more than 1 hr.

Email me and Xudong Tang ([email protected]) if interested, and/or pass this on if you know of social scientists who might be.

How surprising is it that LeBron averaged 27 points, 7 rebounds, and 7 assists in his career but never had one game where he had 27 points, 7 rebounds, and 7 assists?

Joshua Brooks writes:

Listening to a Bill Simmons podcast and the stat came up (not confirmed) that over sixteen years Lebron averaged 27 points, 7 rebounds, 7 assists but never had one game where he had 27 points, 7 rebounds, and 7 assists.

Bill and his guest thought that’s a crazy improbable stat and I guess maybe intuitively it does, but while I’m no statistician I’d imagine the three dimensional nature of the stat would mean it’s not all that in improbable.

But what’s interesting is the category: what is the category of probabilities that seem intuitively (to the average Joe) as improbable but mathematically, are probable? And what are the characteristics of stats that fall into that category?

Hmmm . . . just to do a very crude calculation, suppose that in any game LeBron could score between 17 and 37 points, between 2 and 12 rebounds, and between 2 and 12 assists, with uniform distributions, then his probability of scoring exactly 27/7/7 would be approximately 1/2000. It turns out that he’s played approximately 2000 games, so it wouldn’t be a surprise if he’s ever had a 27/7/7 game but it’s not a surprise if it’s never happened. (I don’t know if it’s really never happened; I’m just relying on a third-hand report here.) I guess we should really bump up the probability a bit here, given that 27, 7, and 7 are in the middle of the distribution, but even if an appropriate probability is 1/1000, it’s still no shocker for it not to happen in 2000 games.

Brooks points to this clip where Simmons describes it as a “crazy” stat, but I agree that it’s not so crazy as all that. Bill Simmons is entertaining and thought provoking, but he’d never be mistaken for a quant.

Frustration when authors dodge criticism

Markus Löcher writes:

I had to think of our discussions on the issue of (not) reproducible research which seems to be a real problem, when I carefully examined the code for this paper, “Stacked regressions and structured variance partitioning for interpretable brain maps,”
published by authors from the machine learning power house CMU.

I quickly discovered a major problem when I noticed that instead of the claimed partial R2 the code computed the marginal R2, which leads to completely different conclusions.

I communicated this to the authors last May and it took them until now finally contact the journal and suggest a small correction, which in my eyes is far from sufficient.

I wrote up my main critique here.

I haven’t read the above paper or critique in detail, so I’m just sharing the story, not trying to make a judgment.

P.S. When following the above link to Löcher’s blog, I noticed this other post about “peeking in AB tests,” so I thought I’d to my post explaining why I think sequential data testing is ok, which is related to my view that type 1 error rate control is not a good thing.

P.P.S. Löcher informs us that the correction has finally been posted.

Why do is it so hard to get a cell phone number with no 0’s or 1’s?

Many years ago—over two decades ago, actually!—I got a cell phone for a family member. I thought it would be fun to have a phone number whose digits spelled out a word (when converting to the 2 = ABC, 3 = DEF, etc., code of the telephone). I had the impression that you could pay the phone company some extra fee and you could pick the number, kind of like a vanity license plate or those businesses that have numbers like CAR-WASH or whatever. It turned out that you can’t do that for a cell phone, or at least this provider didn’t do it. So I said, ok, just give me a number with no 1’s or 0’s, and then I can make a word out of it myself. The person on the line said they couldn’t do that either, but they could give me three numbers and I could pick one. So they gave me three numbers, but all of them had at least one 1 or 0. The person said, no problem, I’ll give you three more numbers. Again, all had at least one 1 or 0. We went through this about five more times and then eventually a number came up with no 1’s or 0’s. So I took it. It didn’t make a great word, but I was able to come up with a reasonable two-word phrase.

I know a lot of people with cell phones, and just about everybody has a number with at least one 1 or 0. I don’t know what’s up with that. There are 7 digits, the first of which can’t be 1 or 0. If the numbers were assigned purely at random, the probability of the number having no 1’s or 0’s would be 0.8^6, which is 26%.

So something’s going on here. Where are all the numbers with no 1’s and 0’s going?

The above question came up in a comment thread. I’m well aware that, back in the old days, there were rules for area codes and phone numbers (all the area codes had 0 or 1 in the middle and none of the three-digit exchanges did), also various reserved numbers such as 411, 911, and 555. But I don’t see any connection between those rules and the current practice where almost all the cell phone numbers have at least one 0 or 1.

Whassup?

OK, here’s a new scam. But I can’t figure out how it’s supposed to work.

The following arrived in my inbox:

Dear Prof. Andrew Gelman

Good day. We are writing with a special academic invitation for Hong Kong this December.

We are an independent academic coordination team based in Japan, and we are helping to coordinate the 2nd Chen Tingzuo Immunological Science Award Ceremony, planned for early December 2026 in Hong Kong. In addition to the award ceremony, the program will include academic lectures and exchanges across different areas of science, medicine and technology. . . .

For this year’s program, we would be very happy to invite you to join us in Hong Kong and give an academic lecture.

The lecture topic is completely open and does not need to be related to immunology. You would be very welcome to speak about your own current research, recent developments in your field, or any subject you feel would be meaningful to share with the audience.

We would be pleased to provide an honorarium of USD 5,000, together with round-trip business-class airfare and local arrangements. A dedicated assistant and interpreter can also be arranged throughout your visit if helpful.

The exact date in early December is still being coordinated. May we kindly ask whether you might have some availability during that period and whether a visit to Hong Kong might possibly work for you?

If there may be some possibility, we would be very happy to share more information and also arrange a short Zoom sometime convenient.

Thank you very much for your time, Professor. We sincerely hope there may be an opportunity to welcome you to Hong Kong.

Warm regards,
**

Obviously this is spam. I have occasionally spoken at medical or public health conferences, but nobody’s gonna fly me to another continent and pay me $5000 to speak on a random topic at such a venue.

The thing that I can’t figure out is . . . how’s the scam supposed to work, exactly? Am I supposed to reply and then they’ll tell me that the free flight and the $5000 check are coming, but first I need to send them some up-front money? That’s too obvious, right?

I’m almost completely sure that the bot that sent the email has no connection to immunology, given that it comes from a weird goacadco.info email address. So, sure, they’re either making up a conference or spoofing a real conference. But, again, to what end? I respond and then they tell me that, by the way, they have a wonderful business opportunity to offer me?

In comparison, the Dr. Ella scam makes a lot more sense to me: I can better envision the sequence of events by which the spammer could eventually profit. This immunology thing . . . I just don’t get it.

The head and the heart

Jackson Lears writes:

Characterising Carter’s views on the nuclear arms race, Luce resorts to a familiar – and misleading – antithesis: the president’s ‘heart’ hated nuclear weapons, while his ‘head’ feared vulnerability to a Soviet first strike. One could just as easily say that the effort to move towards nuclear disarmament came from the head, as the only sane alternative to the fantasy of a winnable nuclear war, and that the fear of a Soviet first strike came from the heart – from the terror that the Soviets were so maddened by ideology as to provoke their own annihilation by America’s weapons.

Good point! I think this problem often arises with head vs. heart arguments, that by labeling one position as the “head,” you’re already rhetorically stacking the deck.

This is kinda related to the One-Way Street Fallacy: Considering only one possibility of a change that can go in either direction. In both cases, the framing of the problem is implicitly doing most of the work.

Overall I disliked Lears’s article—it seems a bit “tendentious,” as the scholars like to say. For example, he writes that CIA backing of jihadist groups “provoked the Soviet invasion of Afghanistan in 1979,” quoting Zbigniew Brzezinski as saying, “We did not push the Russians into intervening, but we did knowingly increase the chances they would do so.” The quote is interesting but I don’t buy the “provoked” thing: Lears is committing the attribution error of assuming that “we” have agency and “they” don’t. Nothing was forcing the Soviet Union to invade a neighboring country. This is not to say that the CIA behavior was appropriate or wise, just that I think Lears is misattributing the blame.

I guess that Lears has a thing about Russia, because later in his article he refers to Putin actions toward Ukraine as “Russia’s concern to maintain friendly relations with its ‘near abroad.'” Friendly relations, huh? Nothing shows friendship like invading and bombing the hell out of a country, I guess. With friends like these . . . . He also writes that Putin “focus on his own nation’s interests,” but I don’t actually see why it’s in his nation’s interests to start an aggressive war. I wonder whether Lears thinks the U.S. government is focusing on our nation’s interests with its bombing campaigns.

Anyway, no need for you to agree with me on the details. The reason for my post is to share my appreciation for Lears’s point about head-and-heart arguments. I put in the rest just to note that I have my problems with his whole article. That’s fine; someone can make a good point within a larger argument that I disagree with.

JAMA rules of engagement on causal and predictive language

Erik, Witold, and I recently published a paper in the Journal of the American Medical Association. This was cool. A day after the paper came out, I ran into someone I know on the street and he said he liked our article. That’s kind of amazing, as usually it seems that nobody reads anything. We’ll have to publish in JAMA more often.

As part of the publication process, we were sent some general reporting requirements per JAMA standards. Here’s the part that will interest you:

USE OF CAUSAL LANGUAGE
Use of causal language (including use of terms such as “effect,” “efficacy,” “cause,” and “x increased y”) should be used only for randomized trials. For all other study designs (including meta-analyses of randomized trials), methods and results should be described in terms of association or correlation (eg, “x associated with an increase in y”) and should avoid cause-and-effect wording.

Avoid use of the words “predict” and “predictive of” for reports of observational studies assessing independent and dependent variables, unless your study design is a diagnostic/prognostic study and has used an appropriate approach to quantifying predictive performance such as (but not limited to) split sampling, cross-validation, or bootstrapping.

On the first item . . . Wow. I wonder if they’d let you write, “Cigarette smoking causes cancer in humans”?

I guess the second item is ok if it’s interpreted broadly enough. For example, Bayesian methods use an appropriate approach to quantifying predictive performance, so I guess that would be included in the “not limited to” category.

Columbia University says, “We believe in difficult conversations at Columbia—in a broad embrace of freedom of expression. It’s in our DNA.”

Claire Shipman, the president of Columbia University, writes:

We believe in difficult conversations at Columbia—in a broad embrace of freedom of expression. It’s in our DNA. I intend to uphold that tradition for several reasons.

First, as a nation, we should be able to tolerate ideas, facts, and opinions that we don’t like. Tolerance is essential to building resilient individuals and a resilient democracy.

Second, it’s our mission. We have a responsibility to guide students toward intellectual rigor, to enable them to grapple with, and examine carefully, ideas different from their own. More broadly, a willingness to test assumptions is essential to our academic excellence in every field.

Third, and perhaps most important, universities—Columbia in particular—are uniquely positioned to help change the national tone. We can guide and model the art of examining complex, potent issues in a deliberate, respectful fashion. . . .

Our instinct might be, in such an era, to shy away from challenging events and speakers. The better course is to model how to do it well . . . Our statutes lay out quite clearly and sensibly that expression, on our campuses, does have limits, in particular as regards harassment, discrimination, or the threat of violence. Peaceful protest, civil discourse, a respectful exchange of ideas—this is our inheritance at Columbia, and I want to ensure that it is also our legacy. . . .

So, as this week begins, and as we continue to debate the affairs of the world and this nation on our campuses, consider this: at times there will be speech that each of us finds disagreeable, even abhorrent—and we still need to protect it. Maybe even seek it out. Such a practice makes us stronger.

I agree. And, although our president did not say so in this message, this policy of protecting and welcoming dissent, even when it is controversial or disagreeable, runs counter to some actions of our university in recent years:

  • The delayed reaction to the sexually-assaulting doctor in the medical school, without any seeming consequences to the people at the university who enabled this behavior.
  • The fake U.S. News numbers, where, again, there seems to have been very little interest by the university administration in finding out who did it and why. As with the medical school scandal, the avoidance of any serious investigation does not seem consistent with the goal of promoting free speech.
  • Barnard College, a division of Columbia University, attempting to cancel the showing of a controversial film on campus. At that time, it seemed that everything was about avoiding offending anybody and about avoiding any controversy—really the opposite of the view that “we should be able to tolerate ideas, facts, and opinions that we don’t like.”
  • Barnard College “mandating that students remove any items affixed to room or suite doors” under the rationale that they “do what we can to support and foster the respect, empathy and kindness that must guide all of our behavior on campus.” Empathy and kindness are fine, but, again, it contradicts the idea that “there will be speech that each of us finds disagreeable, even abhorrent—and we still need to protect it.”

As I noted the other day, Barnard College has also announced a new policy promoting free expression, and I think it’s a good thing.

I completely agree with the president of Columbia that “we should be able to tolerate ideas, facts, and opinions that we don’t like,” and I completely agree with the president of Barnard that “Colleges and universities need to reconfirm our commitment to nonviolent forms of disagreement — even when we are confronted with voices that disparage or dismiss identities and worldviews. This is also a time to foster more disagreement, not less.” I was getting very frustrated with all the happy talk and the official attitude that we shouldn’t be disagreeing. The university—and, of course, the country—has people who disagree with me on important issues, and people I don’t respect at all, for moral, intellectual, or political reasons. And I guess lots of those people don’t respect me either! Which is too bad, but we still have to exist, and we should have the right to express our views, both publicly and privately. We shouldn’t be fired for our nonviolent acts (except to the extent that they interfere with our ability to do our jobs), nor should be discouraged by the university from expressing them.

And, just to be clear, no one at Columbia has ever suggested or even requested that I keep my mouth shut in class or outside of class, including when I’ve criticized the government or the Columbia administration or anything else. I have nothing to complain about on my end, but I do think it was a mistake for them to crack down on dorm room doors, to try to stop the showing of controversial movies, and to keep quiet the bad behavior regarding the fake statistics and the terrible actions at the medical school.

I’m happy about this new policy and I hope the administration follows up by changing the dorm room door policy, apologizing for trying to suppress the movie, and starting and publicizing independent investigations into the U.S. News numbers and the enabling of the sexual assault at the hospital.

Also let me emphasize that by bringing up these earlier cases, I’m not criticizing the Columbia president Shipman, as these things were done before she took office. I also understand the desire to move forward rather than looking back; still, I think that a great way to reinforce the policy of enabling difficult conversations is to revisit some recent examples where the university had unfortunately avoided doing so.

“It’s in our DNA.”

One thing that jumped out at me when reading the president’s email was the statement that freedom of expression, especially in difficult conversations, is “in our DNA.”

My first reaction was the annoyance of the persnickety social scientist: calling it “DNA” is just wrong, and it indeed muddies the waters, as we’re talking about something social, not genetic.

But that reaction of mine was silly. Obviously no one when using “DNA” in that sense is taking it literally; it was a metaphor. And maybe we can gain some insight by taking this metaphor seriously.

For something to be in an organization’s “DNA,” that would imply that it’s not just a transient trait; rather, it has to be in some equivalent of self-replicating code. The idea is that freedom of expression and willingness to confront uncomfortable ideas are not just attitudes that sometimes has existed at Columbia, but rather that they are embedded deeply enough that they will persist. Indeed even if free inquiry and expression at Columbia are suppressed (as with the sexual assault, the fake statistics, the dorm-room censorship, and the attempt to suppress the film screening), the self-replicating aspects of our university’s rules and culture will cause them to return.

And, from that perspective, I do see something in our president’s analogy to DNA. Columbia has for many years celebrated the legacy of the 1968 protests, while recognizing that the other side at the time had legitimate concerns too. That is, Columbia has recognized the value of protest without implying that any particular protest action has to be recognized as being a good idea. We have the University Senate, which dates from that period, as well as generations of new students and staff who have exercised the freedom of dissent—even dissent that some can find disagreeable—and recognize this as a Columbia tradition.

Indeed, this recent statement by the president can be viewed as a consequence of that DNA, an expectation that our university is a place with diverse enough attitudes that there will be people who disagree with and even despise each other but still have the right and, at times, the obligation, to air these uncomfortable disagreements publicly.

Because the alternative, of simulating a nonexistent harmony, would be intellectually dishonest and politically dangerous.

The problem is noise, not N: why increasing the sample size isn’t all that.

Someone was talking with me about the problem with lots of published social science research that’s based on small sample sizes. And I agree that this is a problem! But I think the real problem is noise, not N.

Noise (variation in effect size, variation in outcomes, and measurement error) is more of an issue than sample size because lots of bad studies are so noisy that, to get a large enough sample size to detect a reliable effect, you’d need N = a zillion, and at that point you’d be averaging over such a wide range of conditions that it’s not clear what you’re estimating anyway.

There’s a widespread misunderstanding by which people think that their problems can be solved with bigger samples. I think the erroneous reasoning goes like this: People are aiming for 80% power, but they really have only 50% power, or 30% power, or something like that. If you have 50% power and double your sample size, you’ll have 80% power . . . a win! The error in this reasoning is that often the power is much less than 50%, more like 5% + epsilon. The flip side of this is that if you really are running studies with 50% power or even 30% power, you’re likely to get the right direction on the effect size anyway, so often there’s no good reason to wait on statistical significance.

So, lots of moving parts here: There’s the underlying effect size, the variation in effect size, the variation in the measured outcomes, and researchers’ expectations when setting effect sizes and interpreting results.

Digital twins version 2.0: The skeptics speak up

This is Jessica. Several years since the earliest research papers on the topic came out, the idea of using LLMs to simulate human behavior like survey responses or experiment outcomes (“silicon sampling”) is hitting mainstream. People tend to bring this up in conversation with me, probably because I’ve written about validation and taught a course, so I get to hear a lot of other researchers’ views. I would put the majority of takes I encounter in either the more agnostic, “Somebody should probably explore how much we can do with them” and/or “I’m pretty doubtful it’s going to live up to the hype” camps.  

One way to use silicon samples is digital twinning, where simulations are conditioned on considerable information about specific individuals, and expected to predict the responses of those specific individuals. Early papers on digital twins sing their praises, showing for example that they can reproduce people’s responses more than 80% as well as people replicate their own responses (e.g. here or here). But we’re now seeing a more critical empirical push, which uses novel tasks and topics and more careful design of metrics and baselines for comparison to gauge how much conditioning on more individual-specific information improves prediction. 

In the last month, there have been a few noteworthy data points. Pew Research put out a report describing what they learned from comparing digital twins implemented by giving Claude Opus 4.6 detailed 2025 survey responses and demographic persona information for people who responded to several more recent survey waves. They compute the average absolute error in percentage points for each multiple choice survey question (i.e.,the average of the absolute differences in the proportion of LLM versus human respondents selecting each answer option), and find it averages 12 percentage points over the 300 survey questions they looked at, with 28% of the questions having error over 15 percentage points. They observed several kinds of biases that have come up in other research comparisons: less variation in LLM responses compared to human (with some answer options never being picked by the LLM), stereotyping (like predicting that 97% of Hispanic adults are at least somewhat likely to follow the World Cup; in reality its only 43%), overestimating human accuracy on factual questions, and mispredicting sentiment on current events like data centers and face coverings on ICE officers. They stress how hard it is to predict how far off the LLMs will be on a given survey question.  

This list of biases closely resembles those proposed by Tianyi Peng et al. in one of the most comprehensive evaluations of digital twins to date, recently published in Science Advances: Digital Twins as Funhouse Mirrors: Five Systematic Distortions. They collect a new set of 19 preregistered studies involving 164 outcomes, and find that overall, there is only weak improvement over base models and relatively low correlations with human behavior (on the order of 0.2). 

I like this paper a lot, in particular the comparisons they do to simpler demographic prompting and base models. The intuition that has been sold by researchers and companies in the space alike is that with rich enough information about individual people, we can predict their behavior with high accuracy. To make the twins approach worthwhile, we want to see that twinning specific individuals leads to responses that predict variation between individuals better than the demographic baseline. However, some of the early papers advocating for digital twins evaluated them by looking at measures where a model can do well simply by being a good predictor of the average response. For example, you can look at the correlation between each individual’s real and twin response over a set of likert-style survey questions, but a model that does a good job of predicting on average which questions will get higher responses may appear to do well by this measure, even if it does a poor job of sorting individuals for any particular question. 

Instead, Peng et al. look at correlations between human and twin responses for a single outcome at a time, and compare to demographic personas only as well as base models, to ask how much additional predictive power the twin approach provides when it comes to sorting people into those that will respond with higher versus lower values. They find that while there’s a boost in average correlations (from 0.08 with base models to 0.15 with demographic personas to 0.2 with twins), overall the correlations remain modest, and much more modest than the correlations over questions that other papers have reported. Hopefully it encourages more care in future papers when it comes to what measures actually support the claims being made. I do think behavioral researchers and others interested in this space are gradually changing their mental models to conclude that LLMs may do a decent job of predicting population averages for well-represented populations, but they have a long way to go when it comes to modeling individual heterogeneity. There are several other papers along these lines that I haven’t had a chance to look closely at. 

How damning is this for the approach itself? I think it remains hard to say, though it certainly suggests skepticism is warranted at this point in time. The reasons I wouldn’t write digital twins off completely are 1) it is still early and models continue to improve and 2) prompting is often not the best way to recover the predictive information a model has–you can do better at opinion prediction by using model internal representations (see, e.g., this). On the other hand, ultimately it comes down to limits on how predictable individual behavior is, and while people are very predictable in lots of ways, we are also always exercising our agency and changing in subtle ways that interact with the world, which even we ourselves can’t necessarily understand or predict. So I don’t think reaching AGI is synonymous with being able to predict the majority of individual opinions and actions. I’m reminded of Matt Salganik et al’s work on the Fragile Families Challenge, predicting life outcomes from thousands of variables, where the best of the 160 teams’ models explained only about 20% of the variation on the most predictable outcomes, despite the enormous amount of data. 

What I will say with certainty is that silicon sampling has attracted some very good salespeople. I expect a lot of money will be made before anyone abandons the idea.

Thinking more seriously about the design of exploratory studies: A manifesto

I came across this post from 2016 which I still think is important.

The idea is that there should be some guidelines for experimental design, not just for “confirmatory” studies that produce estimates, standard errors, predictions, hypothesis tests, Bayesian inferences, etc., but also for exploratory research.

As Ed Hagen put it:

Exploratory studies need to become a “thing.” Right now, they play almost no formal role in social science, yet they are essential to good social science. That means we need to put as much effort in developing standards, procedures, and techniques for exploratory studies as we have for confirmatory studies. And we need academic norms that reward good exploratory studies so there is less incentive to disguise them as confirmatory.

Here’s what I wrote, a decade ago:

Suppose you know ahead of time that your theories are a bit vague and omnidirectional, that all sorts of interesting things might turn up that you will want to try to understand, and you want to move beyond the outmoded Psych Sci / PPNAS / Plos-One model of chasing p-values in a series of confirmatory studies.

You’ve thought it through and you want to do it right. You know it’s time for exploration first and confirmation later, if at all. So you want to design an exploratory study.

What principles do you have? What guidelines? If you look up “design” in statistics or methods textbooks, you’ll find a lot of power calculations, maybe something on bias and variance, and perhaps some advice on causal identification. All these topics are relevant to data exploration and hypothesis generation, but not directly so, as the output of the analysis is not an estimate or hypothesis test.

So I think we–the statistics profession–should be offering guidelines on the design of exploratory studies.

An analogy here is observational studies. Way back when, causal inference was considered to come from experiments. Observational studies were second best, and statistics textbooks didn’t give any advice on the design of observational studies. You were supposed to just take your observational data, feel bad that they didn’t come from experiments, and go from there. But then Cochran, and Rosenbaum, and Angrist and Pischke, wrote textbooks on observational studies, including advice on how to design them. We’re gonna be doing observational studies, so let’s do a good job at them, which includes thinking about how to plan them.

Same thing with exploratory studies. Data-based exploration and hypothesis generation are central to science. Statisticians should be involved in the design as well as the analysis of these studies.

So what advice should we give? What principles do we have for the design of exploratory studies?

Let’s try to start from scratch, rather than taking existing principles such as power, bias, and variance that derive from confirmatory statistics.

– Measurement. I think this has to be the #1 principle. Validity and reliability: that is, you’re measuring what you think you’re measuring, and you’re measuring it precisely. Related: within-subject designs or, to put it more generally, structured measurements. If you’re interested in studying people’s behavior, measure it over and over, ask people to keep diaries, etc. If you’re interested in improving education, measure lots of outcomes, try to figure out what people are actually learning. And so forth.

– Open-endedness. Measuring lots of different things. This goes naturally with exploration.

– Connections between quantitative and qualitative data. You can learn from those open-ended survey responses—but only if you look at them.

– Where possible, collect or construct continuous measurements. I’m thinking of this partly because graphical data analysis is an important part of just about any exploratory study. And it’s hard to graph data that are entirely discrete.

I think there’s room for something longer and more systematic on this topic. It’s important.

Getting better causal inferences by using more information (my talk at Stanford this coming Wed)

It’s the political economy seminar at Stanford business school, Wed 14 Oct at noon in room M109:

Lagged regression, difference in differences, two way fixed effects, regression discontinuity, placebo controls: these are different methods used when estimating causal effects from panel data. We’re working on a general approach using multilevel regression and poststratification (MRP) that aims to get the best from all these approaches. We demonstrate with a range of examples, including observational studies in economics, political forecasting, and controlled experiments in biology. This is joint work with Jori Korpershoek and others.

Come and ask me some tough questions!

The talk will be pretty open-ended. We have lots of ideas and some successes (for example, the chickens paper) but it’s mostly open questions and promising research directions. Should be fun.

I won’t be talking about this paper or this paper at all—that’ll have to wait until I’m invited to speak in the English department.

What’s the deal with the RMET (“the Reading the Mind in the Eyes test”)?

Dan Luu writes:

Have you ever looked into the RMET? [This is something called the Reading the Mind in the Eyes test, which I’d never heard of until receiving this email. — AG] I’d like to get some data to replicate a result. I have a survey here.

The story, as it relates to me, is that a friend of mine was hanging out in a cabin with a bunch of friends. A friend of theirs, who’s a psychologist, has them take this test. After they take the test, the psychologist reveals that neurotypical people complete the test in N minutes vs. M minutes for autistic people. People sort of can’t believe it, do all neurotypicals really finish the test that quickly? That’s super fast! But it’s plausible that everyone in the room is autistic, so it’s not totally unbelievable.

My friend then has two of his most normal friends take the test and they’re actually slower than he is. They have a discussion about social interactions, etc., and they conclude that they’re autistic, which is surprising/interesting news to them.

Ok, that’s an interesting story so far. I got that part of the story from this friend, Then, a week later, he messages me about this thing again. He had some more normal seeming people take the test and none of them finished the test under the “neurotypical” threshold, so that’s a bit suspicious.

I look into it a bit and, according to some source, neurotypical finishing times are 2-3 minutes, with > 3 minutes being autistic. I can’t find any sort of citation for this, whether it’s a paper or someone putting the data on a website for anything like that. It’s just a site where they have the test online and make the claim that some scientists figured this out. There’s also some study with schizophrenics where the 59 “healthy controls” took 5.45 +/- 2.58 minutes. This makes the 2-3 minutes for neurotypical folks vs. 3+ minutes for autistic folks thing seem like it might be fake?

Also, on looking into the RMET further, the untimed version also seems pretty fake for a bunch of reasons, one of which I’d like question-level data for.

One thing that seems fake is that some online places you can take the test report neurotypical vs. autistic thresholds like the time-based threshold, but on looking up what the distribution of results is, it’s pretty clear that the difference within a group is much larger than the difference between groups, so no such threshold-based criteria can really make sense. Also, it turns out that Asian folks score worse than white folks on the test, unless you make a race-normed RMET, and just eyeballing the difference, the difference there is something like the order of the alleged difference between autistic and non-autistic folks.

For me, on the official test, the facial expressions that took me the most time (and I made errors on) were some of the photos of women who look like models or actresses. I think that’s because I don’t normally interact with people who look like that (vary manicured eyebrows, totally flat skin with no wrinkles but not a young child, etc.), so the faces aren’t super readable to me, especially in combination with the relatively subtle things some of the questions are trying to have people read. My guess is I’m not the only person like this and a lot of other people will miss the same questions, but maybe not.

Anyway, do you have suggestions on how to go about this? My current plan is that I’ll just tweet out the survey after some testing to make sure the survey works, but the set of my twitter followers or whoever sees it from it being retweeted is not going to be very random, but I don’t know that I have a good way to fix this problem.

And Dan shares a very small bit of evidence:

A friend of mine tried it and, while doing it, said she couldn’t make heads or tails of any of the expressions, but then scored a 25 when the average score from totally random selection would be 9. She was surprised by how many she got right despite feeling like she was picking randomly.

Interesting. Try it out for yourself!

What about my reaction?

I tried it, and . . . I didn’t get the point at all. They just looked like eyes: I didn’t see any facial expressions at all. This might just be me: I have Tourette’s syndrome and just about never look at people in the eyes. I almost always look away. It’s not that I never look at people in the eyes; it just feels unnatural to me to do so, and, when I do look someone in the eyes, I don’t do it for long. So, unlike most people, I don’t have a lifetime of practice looking at people in the eyes. I remember back in elementary school a teacher getting mad at me for not looking her in the eyes, and I was thinking, Hey, just tell me what you want to say, I’m listening to your words.

So, for me, it wasn’t a matter of finishing the test in X minutes. I couldn’t do it at all. I gave up after the first two images. The funny thing is, I know that people can get emotional information from eyes; I’m even a coauthor of a paper on the topic. But this is the first time I’ve ever tried to do this myself, and I just can’t. I’m not autistic, I have no problem with emotions, it’s just the eyes thing.

P.S. A reader points us to this recent report: Higgins, W. C., Savalei, V., Polito, V., & Ross, R. M. (2026). Reading the mind in the eyes test scores demonstrate poor structural properties in nine large non-clinical samples. Assessment, 33(2), 204-220.

Our continuing struggles with the Monster model

tl;dr. We’re still stuck.

It was 30 years ago that Frédéric Bois, Jiming Jiang, and I published our paper on hierarchical modeling for toxicology. It was a successful paper–it’s been cited over 400 times! It won a major award! More important for me was that working on this project gave me a much better understanding of several different areas of Bayesian inference and computation. I think the greatest impact of this paper has been on informing my thinking, giving me a perspective which has then spread through my research articles, textbooks, and software. I don’t know that it our approach is used much in toxicology, as it took a lot of work just for that one example.

The example is a study of the population toxicokinetics of tetrachloroethylene. Here’s the applied paper with details of the model, and here’s the statistics paper with an explanation of how we did the Bayesian modeling and inference.

The data come from an experiment with 6 people, and for each person there is a multi-compartment model set up as a differential equation with about 15 parameters, and then we had a hierarchical model to allow these parameters to vary by person. We did the computation using Gibbs and Metropolis with a differential equation solver in there too. You can check out the above-linked paper to see how we set up priors and checked the fit of the model to data.

Back in 1993 or 1994 or whenever, it took a lot of work to fit the model. Frédéric programmed it in an C program he wrote called mcsim. But finally it did work, the model fit the data and made reasonable predictions, and we published the papers. As I said, it was a lot of effort for just one toxin, but the work served as proof of concept for the use of Bayesian methods for a complicated model that could not be fit using standard approaches given the sparse data that were available.

I put the example in Bayesian Data Analysis, and then eventually I wanted to fit the model in Stan. When using an example for teaching, it’s good to be able to run the code directly, rather than just pointing to graphs in a published paper.

But it wasn’t so easy to fit the model in Stan! It seems that there are some difficulties with identification of the parameters; perhaps this is one of those cases where computation works ok with a crude algorithm (random-walk Metropolis), but then when you switch to a better algorithm (Hamiltonian Monte Carlo), complexities arise. It’s like picking up a rock and seeing all the worms underneath.

We had some discussion of the problem on the Stan Forums a few years ago. As part of that conversation, Charles Margossian wrote:

It has a lot of moving parts: a stiff ODE that is highly sensitive to parameter values, sparse data, a low number of patients (which makes it it difficult to estimate inter-individual variability), hierarchies across multiple parameters, etc. The workflow is complicated by the fact each model fit takes several hours, even after firing up the cluster and parallelizing within chains.

I [Charles] fitted simplifications of the model and got reasonable results. Fixing one or two parameters, or the population variance greatly simplifies the problem. The full model returns divergent transitions and I haven’t found the right parameterization. I tried monolithic centering and non-centering, and things in between. I now suspect the problematic geometry may not be only due to the interaction between population and patient parameters. It’s indeed possible to have a funnel between population parameters, in which case no parameterization can help you. Divergences aside, I don’t trust the estimated parameters, because there are somewhat inconsistent with the priors which were very carefully constructed. Truncating priors, as was done in the original paper, limits our ability to reparameterize the model. When using truncation, I find a lot of the probability mass concentrates at the boundary (note that on the unconstrained scale, you actually have infinite volume near the boundary). All that said, you shouldn’t need to truncate the priors, because the truncation occurs at the tail of the prior distributions.

The original paper didn’t use HMC, so they didn’t have access to diagnostics such as divergent transitions. You can also imagine how an accept-reject sampler will interact differently with a truncated distribution.

To sum up, I agree this would make a very good case study and writing it has been my ambition for some time. But it’s a challenging model to fit, with behaviors I don’t fully understand (yet). There exist other examples of hierarchical ODE models, built and used with Stan, which can be considered, if the goal is to write a case study. But yes, I think we have a lot to learn from the Monster model.

Lots more in that thread, including this plea from me:

I’d be soooo happy if someone could get the Monster model running in Stan. It’s a great example but it would be so much better if it were runnable. Also it’s buggin the hell out of me that we could it just fine using Metropolis 25 years ago but now we’re getting stuck when using a much better algorithm on computers that are a zillion times faster. So I’d really really really like to get this one straightened out. Anyone who can do this, I promise you a free homemade loaf of bread, an advance look at some blog posts 6 months in the future, and pretty much whatever else you want from me.

We’re still not there yet, and I don’t know the full story. The above offer still stands. Here are the data and some code from Charles.

I’m sharing this example for two reasons:

1. I’d still like to fit the model in Stan. Not just to show off Stan, but because if the model is in Stan, it’s portable. Other people can use it for other, similar, problems. It would be a good teaching example (even though now it’s too late for it to be included in the Bayesian Workflow book). And, given that we did have problems with the computing, it would be good to know what it takes to get the model to work.

2. It’s good to have examples of our fallibility. Of course I’ll share the successes (see here) but the struggles are important too.

P.S. to Bob: I hope you appreciated the careful use of capitalization to convey meaning in the title of this post.

Survey Statistics: 2 different uses of simulated data

Last month Andrew wrote:

I use simulated data to understand how a model works; it can be viewed as a form of mathematical analysis, a way of working out the implications of a set of assumptions.

I don’t use simulated data to learn about the external world. For that, I need to take measurements of reality.

Let’s start with the latter, which concerns using chatbot-simulated survey responses.

Simulated data to learn about the external world:

Andrew wrote: “the chatbot is trained on the past, but the usual reason for surveys is to learn about changes”. Pew Research agrees: “Survey-taking is best left to humans”. And a year ago we saw that Thomas Lumley agreed: “It [Claude or ChatGPT] will probably do this better than I could, but it’s not magic.”

That said, Andrew suggested using the simulated response as an additional adjustment variable. Jessica Hullman wrote last year about a suggestion from Broska, Howes, and van Loon (2025) that uses LLM predictions as auxiliary data in something like a survey regression estimator. Andrew of course suggests Multilevel Regression and Poststratification (MRP) to incorporate auxiliary data like LLM predictions.

Now back to the first use of simulated data.

Simulated data to understand how a model works:

Andrew has written lots about this. I teach it in my course with Dr Arjun Potter at the Nelson Mandela African Institution of Science and Technology (NM-AIST). Our example is Potter et al. (2026), who study the effects of drought, fire, and herbivory on growth of various acacia tree species. In our course, we simulated data ahead of real data collection to understand the model, e.g. how block effects change the precision of the fire effect. Students understood that this exercise does not replace real data collection. We are hoping to teach these ideas again in the new year (funding dependent). For the past three iterations, see “Hand-drawn Statistical Workflow at Nelson Mandela” and “Survey Statistics: connections to experimental design”.