My exchange with Yann LeCun about killer robots

This was from 2022. Yann LeCun:

People being scared to death of technological progress is nothing new either.

There is a very long list of retrospectively-hilarious fears of every new technological or cultural progress.

An interesting case is the printing press, which the catholic church saw as potentially destroying the fabric of society.
And that, it did.
And we are all better off for it.
https://newsletter.pessimistsarchive.org

Me:

I have to admit, I’m scared of drones flying around with machine guns.

LeCun:

I’m scared of machines guns in the wrong hands, which includes those of average citizens.
I’m scared of machine guns mounted on drones controlled by unsavory characters.
But I don’t really mind them in the hands of the well-trained military of a liberal democracy for the defense of freedom.

Me:

There are two questions here:

1. Given that the technology exists to make drones flying around with machine guns, what should our government do about it?

2. Am I scared of drones flying around with machine guns?

The answer to question 1 is complicated. I guess that our government has many options, including research and development on more effective drones flying around with machine guns; research and development on anti-drone weapons; and negotiating treaties to limit research, development, and deployment of these devices.

As to question 2: Yeah, I’m scared of drones flying around with machine guns! Partly because the “bad guys” might use them in terrorism, and partly because “the good guys” might use them in terrorism. The U.S. has been a liberal democracy for a long time and that hasn’t stopped its military from terroristic acts from time to time.

Anyway, yeah, it’s scary to me. Sometimes I see little drones flying around in the park. If I thought they might have machine guns on them . . . jeez!

I guess it also depends on what you call a “liberal democracy.” Considering countries that have recently started wars, Russia and Israel don’t count: Russia has elections but it seems to be pretty much understood that they’re controlled by the government, and Israel has what’s called a herrenvolk democracy, which is a liberal democracy only for the dominant ethnic group. The U.S. is a liberal democracy with a well-trained military, but it’s not always used “in defense of freedom.”

I guess it’s that last bit—“in defense of freedom”—that’s the sticking point. Arguably the ability to wage war with machine guns mounted on drones will enable military leaders, whether or not they’re from liberal democracies–to act aggressively, not in defense of freedom at all.

Another way to look at it is that “unsavory characters” are everywhere. And giving people the power to kill remotely can perhaps bring out the unsavoriness of some characters.

In any case, LeCun is a computer scientist and I’m a statistician, and neither of us will be in charge of government policy. Which I guess is a good thing. At least, I know it’s a good thing that I’m not in charge of anything.

Real political reporters use data journalism techniques to catch fakers at “Wedgewood Polls”

Mary Radcliffe, Elliott Morris, Cooper Burton, and Fiftyplusone report:

On Tuesday, Oct. 20, FiftyPlusOne reached out to Wedgewood Polls, a self-professed polling firm, with a request for comment on an investigation we had conducted into their alleged survey research. . . .

We reached out to them to ask who they were and confirm the methodology statement on their website. We were given two names, Jack Sims and Andrew Prater. We were able to independently confirm that Sims had previous experience in the polling industry and reviewed their methodology statement in detail. It is not atypical for a pollster to leave an existing firm, form their own, and self-fund some surveys to build reputation; we had no reason at the time to believe this case was any different. Wedgewood operated as if it were a legitimate polling firm, including collecting money from donors to conduct crowd-funded polls, and sharing raw survey data (which we now have confirmed was fraudulent) with FiftyPlusOne on Oct. 19, 2026.

How did Radcliffe et al. figure out Wedgewood data were fake?

First, in FiftyPlusOne’s data from this vendor [Pollfish, an online survey platform], the ID variable for a person appears as a roughly even mix between a standard string of numbers (e.g., “1234567890”) and a string of numbers prefixed by a hyphen (e.g., “-1234567890”). Second, the transaction identifier correlates almost 1:1 with the time a person submitted their survey. We confirmed with Pollfish that this is the expected behavior of these columns, and verified that data from another pollster who has used the platform also produces patterns that match ours.

However, the respondent and transaction IDs recorded in survey data that Wedgewood provided to FiftyPlusOne do not match these expected patterns. Wedgewood told us it conducted a recent poll of the Mississippi Senate race using the same survey platform that we tested, Pollfish. However, in Wedgewood’s data, the respondent and transaction IDs do not behave the same way they do in FiftyPlusOne’s data, or in data from the third-party pollster we consulted. Additionally, they do not comport with the descriptions of these columns given to us by the survey platform.

These are just two of many factors that make FiftyPlusOne suspicious of the veracity of the polls being published by Wedgewood.

Maybe there’s an innocent explanation . . . OK, maybe not:

We detailed these concerns to Wedgewood and, they publicly responded indicating that their polls were indeed fake.

Where did the suspicion come from? Radcliffe et al. tell the story:

Last week, we received a credible tip from a trusted source that Wedgewood Polls might be faking data. We had received tips about Wedgewood in the past, but this particular tip had something crucial that other reports didn’t: evidence. So we asked Wedgewood to provide us with the raw data from their most recent survey, a Mississippi poll conducted Oct. 12 – 14, 2026 . . . We were already wary of this survey. The poll was released just hours after a prominent race rater, Sabato’s Crystal Ball, had changed its rating of this race from Safe Republican to Likely Republican (somewhat of a surprise to elections watchers) and showed Hyde-Smith up only 4 points over Colom. That’s a notable coincidence.

When we got the raw data from Wedgewood, we found that it was surprisingly clean. For example, the raw, unweighted data was 67 percent white and 31 percent Black, fairly close to what might be a reasonable statewide estimate of around 60 percent white and 37 percent Black. The design effect for the poll, a measure of how much the pollster had to weight the data in order to get a representative sample, was 1.18, which is fairly low. . . .

It’s possible, however, to achieve lower design effects by using quotas or other techniques to construct a more representative sample at the time of recruitment. . . . We double-checked with Wedgewood that they didn’t use quotas to create their sample, which could explain the low design effect. A quota is a mechanism by which a pollster can constrain the sample, to ensure the panel provider returns a certain minimum number or percentage of respondents in each demographic group, and appears to be the only way Pollfish allows researchers to control the demographics of the sample during fielding. They said they didn’t, which made the low design effect especially unusual. But we were still left with a question: Was there anything that actually proved the data was fabricated?

That’s when we decided to try to replicate their survey, to the last detail. . . .

Lots more information at the link.

Radcliffe et al. then step back to survey the bigger picture:

It’s getting easier to conduct low-quality polls — or fake them altogether. . . . the wrongdoing we have uncovered on the part of Wedgewood Polls is just the most recent in a series of cases of pollsters manipulating and fabricating data. . . . AI-powered tools for generating content such as websites, PDFs, and spreadsheets have made it easier for those willing to act in bad faith to produce potentially bogus documentation of results that match the quality of those produced by legitimate research firms.

This reminds me of our recent post, It’s fine to use computer-generated survey responses—if you don’t care about the data anyway!

In this case, “Wedgewood Polls” responded with a social media post describing themselves as “an experimental project to demonstrate the vulnerabilities, lack of oversight, and overall low barrier to entry in the modern political polling industry.” Maybe so, but it’s kind of suspicious that they happened to release this statement on the very day that they got caught.

I continue to think there are too many polls, even just counting the legitimate ones. As I wrote back in 2004, and it’s so much truer now:

The U.S. is over-polled. You might have noticed this during the recent election campaign when national polls were performed roughly every 2 seconds. . . . My complaint is not new, but this recent campaign was particularly irritating because it became commonplace for people to average batches of polls to get more accurate estimators. As news consumers, we’re like gluttons stuffing our faces with 5 potato chips at a time, just grabbing them out of the bag.

There are two problems with all these polls. First, they’re a waste of time and money. Millions of people wasting time answering polls. Instead they could be planting flowers, cooking dinner, enjoying themself in some way. The second problem is that all these useless polls have decreased people’s willingness to participate in surveys (for example, of public health) that could have general benefit.

The problem with horserace journalism is: first, journalism resources are limited, and effort that goes into horserace journalism could crowd out more valuable journalism; second, it can degrade public discourse.

The connection of too many polls and too much horserace journalism is that they feed on each other: the polls provide headlines for horserace journalism, and horserace journalism motivates more polls.

Throwing fake polls into the mix makes things that much worse. Good job by Radcliffe et al. to catch these particularly noxious offenders.

Survey Statistics: textbooks adding non-probability sampling chapters

Last week I asked for your favorite survey statistics textbooks. Thank you for many fun recommendations here and on Bluesky ! I am compiling a reading list for after the election. Here’s some not-blue-sky on the AT in TN:

Ben Schneider and Winston Lin share my love for Lumley 2010. I agree that it’s fun and a second edition would rock.

Preview

Some recommended books added sections on non-probability sampling in their new editions:

Lumley 2010 already includes material applicable to non-probability sampling, see e.g. “2 flavors of calibration”. If there is a second edition maybe it will follow the books above and add a chapter about non-probability sampling ?

(See “probability samples vs epsem samples vs SRS samples” for more about the term “non-probability sampling”.)

CmdStanR 1.0 release candidate is now available!

Jonah writes:

I’m happy to announce that a release candidate for CmdStanR 1.0 is ready and available for testing! This will be by far the biggest release since the initial release of the package. One of the main changes is to how models are compiled, so we’d like to get people to try it out and provide feedback before we release it. If all goes well, we plan to make the official 1.0 release in a few weeks.

For users of brms, instantiate, and rethinking, there are either releases or branches available that are compatible with the release candidate (see below). If you are a developer of another package that uses CmdStanR and have questions about updating your package please reach out.

Install it from the v1.0 branch on GitHub in a fresh R session:

remotes::install_github("stan-dev/[email protected]")

(or pak::pak(“stan-dev/[email protected]”)). Then check packageVersion(“cmdstanr”). The release candidate should show up as version 0.99.0. The final release will have version 1.0.0 when we merge the v1.0 branch. There is also a version of the CmdStanR website for the release candidate that contains the updated documentation, vignettes, and release notes: R Interface to CmdStan • cmdstanr.

The release candidate requires R 4.0 or later and CmdStan 2.37.0 or later. If your CmdStan is older, install_cmdstan() will install the latest version. . . .

Lots of stuff here that will make Stan run faster and more smoothly. Click on the link for details.

“In the belly of the whale: Portraits from a fractured Israel”

The above-titled book, by the journalist Ruth Margalit, is a collection of profiles of political and cultural figures published between 2015 and 2026. It’s interesting and I learned a lot about these people and their difficult choices they need to make. Some of the people Margalit interviews are on the left, politically (which is where Margalit stands) and some are on the right, but in all the interviews you get a sense of the constraints under which they’re operating.

I’d say that the underlying theme of this collection of essays, even though Margalit does not say so explicitly, is Israel’s herrenvolk democracy—that’s the term used by some political scientist to describe a political system such as apartheid South Africa or the pre-civil-rights U.S. south that is democratic for the dominant ethnic group. (For two takes on Israeli democracy, see Smooha, 1997, and Sa’adi, 2000).

A term such as “herrenvolk democracy”—or, for that matter, “democracy” itself—is not precisely defined, for many reasons: no country gives equal rights and powers to all its residents; the term “ethnic group” is not always clear (for example, was Britain a herrenvolk democracy before they introduced universal suffrage?); there will always be some rights and privileges outside of democratic control; there are different sorts of representative democracy; etc etc.

That said, central to Margalit’s book is the tension between the democracy and political freedoms enjoyed by the dominant ethnic group in Israel/Palestine, and the systems that govern others within the territories. This has been a commonplace ever since the founding of Israel and has been stressed repeatedly by observers both inside and outside the country—just for example, I happen to have a book of essays by Canadian novelist Mordecai Richler containing an article from 1962, “This Year in Jerusalem,” which concludes:

The souvenir shop I stopped at in Nazareth dealt in the usual bogus articles, water from Mary’s Well and so forth. The enterprising Arab proprietor, however, also sold little bags of earth; half of them labelled ‘Earth from the Holy Land, Nazareth’, the printing superimposed over a cross; the others reading ‘Earth from the Holy Land, Israel’, a blue Star of David fixed above. I laughed. The Arab laughed. And it was with this shrewd irreverent Arab that the land of Israel came full circle for me. This Arab’s gift for survival and self-evident humour seemed profoundly Jewish to me, more Jewish than the sabra. I could identify with him.

A lawyer I met in Jerusalem told me that he had served on the Gaza Strip when Israel had occupied the zone in 1956. ‘Don’t forget they’ve been there for fourteen years. Many have died out and others are not proper refugees at all. They were born in Gaza, they’ve never even seen Israel. The figure of one million is an inflated one too. When a man dies they don’t hand in his ration book, but go on claiming his food.’

‘But surely,’ I said, ‘if the Jews are entitled to come “home” after two thousand years then the son of an Arab refugee is a Palestinian too?’

‘All right. Conditions in their camps are deplorable. However, the conditions I lived under in Dachau were worse.’

That was from an outsider. Margalit is an insider, writing 60 years later, mapping some of the fault lines of this political and social arrangement.

“If you decide to write about this paper in your blog, please don’t mention my name. It is important to me to remain anonymous. . . . exposure might harm me or my students.”

So, this came in the email one day:

I am a social psychologist, and I was asked to review a PNAS paper . . . I don’t consider myself an expert in statistics, but I could easily detect three types of issues, which according to your blog are not uncommon in this journal:

– Overstatement of findings
– The journal’s willingness to accept very weak findings (e.g., marginal effects, deviations from preregistration on the key analysis)
– Dishonest reporting: presenting exploratory analysis as if it was preregistered, presenting preregistered hypotheses that were not confirmed as if they were exploratory, deviation from preregistration that is not acknowledged (e.g., not using an important preregistered covariate that should have been used). . . .

In my understanding, the paper creates the impression that a certain intervention . . . is effective, while in fact the evidence suggests otherwise. Personally, I spent a lot of time, effort, and money of funding agencies on an intervention that didn’t work, and I think that it’s important that as a field we move forward and abandon ideas that are wrong. . . .

My correspondent continued with details of various analyses in the paper that were not consistent with the preregistration and then added:

If you decide to write about this paper in your blog, please don’t mention my name. It is important to me to remain anonymous. . . . exposure might harm me or my students.

My reply: Yes, without looking at all the details, the general impression you give reminds me of other examples of this sort, unfortunately.

I’m not linking to the article here, partly to preserve anonymity but mostly because the point here is how commonplace this all is.

“Death and Taxes: Taking Causal Inference into the Real World”

Nandita Mitra is giving a webinar this Tuesday at noon with the above title and the following abstract:

Statisticians develop methods with the hope that they will ultimately inform decisions and improve the world around us. But taking causal inference into the real world requires more than choosing the right estimator: it requires identifying the question that matters, engaging deeply with its context, and communicating statistical reasoning to very different audiences.

In this talk, I will draw on two experiences in which causal inference took me well beyond traditional statistical applications. I will first discuss my experience serving as an expert witness in a death penalty case involving racial disparities in jury selection, where ideas including confounding, mediation, and individual- versus population-level effects had to be translated into compelling and understandable arguments in a court of law. I will then describe my involvement in evaluating Philadelphia’s excise tax on sweetened beverages, where cross-border purchasing led us to confront interference and develop methods to better inform policy decisions.

Through these examples, I will reflect on how statisticians can seek opportunities to bring their expertise to consequential real-world problems, and on the importance of collaboration, communication, and stepping outside our usual academic boundaries. Most importantly, you don’t have to choose between being a methodological statistician and having an impact outside statistics.

This looks interesting. Click on the link above to register for the talk and get the zoom link.

“I write to seek your help and advice regarding a published article that my group and I believe is invalid and should be corrected or retracted.”

David Allison writes:

I write to seek your help and advice regarding a published article that my group and I believe is invalid and should be corrected or retracted. Despite our concerns, the editor seems to be unwilling to accept responsibility and to be responding to us in a way that we perceive as unethical.

You may wish to see a full account on this matter including the Letter to the Editor (LTE) that was rejected by the journal [Di Molfetta et al LTE (short version) Submit.pdf (attached)], our email exchange with the editor following its rejection [Diabetes Research and Clinical Practice Editor Email Exchange.pdf (attached)], and our email exchange with the authors and editor prior to submitting the LTE [Di Molfetta Email Exchange.pdf attached)], but a succinct summary of the events are provided below for brevity:

07/30/2023: We identified a paper as having errors related to not properly analyzing cluster randomized trial data.
08/08/2023: We contacted the original study authors and requested the deidentified data to reproduce the published results.
08/21/2023: The authors requested clarification for the reason for the data request.
08/21/2023: We provided clarification for the data request.
09/22/2023: We prompted the authors to reply to our prior communication.
09/30/2023: We included the editor in the email exchange requesting that they uphold the journal’s data sharing policy.
10/14/2023: We prompted the editor to reply to our prior communication.
10/14/2023: The editor replied and asked for clarification for the reason for the data request.
11/03/2023: We clarified that we requested the data to reproduce the analyses and re-analyze the data using the correct statistical methods for the study design. We also provided a summary of the errors we identified.
11/23/23: The authors replied with their rebuttal to sharing the data, rationalization for their errors, and rationale for not correcting them.
04/29/24: We submitted a letter to the editor to the original journal.
05/01/24: The manuscript was rejected on the basis that our “paper cannot be assigned sufficient priority to allow publication.”
05/08/24: We wrote to the editor directly to reconsider publication of our letter.
05/09/24: The editor responded that the journal does not wish to be engaged in a legal issue and that the matter will not be entertained by the journal any longer.
05/10/24: We responded to the editor to clarify that we simply want the journal to publish our letter and had not proposed any legal action.

The editor has not responded to our last communication as of 05/21/24.

We believe that the most recent response from the editor is patently unethical and incommensurate with the publishing standards of Elsevier, guidelines of COPE, and generally accepted norms in the field of science.

We have two requests:

1. Do you have any advice for us on how we might best pursue this issue further.

2. Would you be willing to blog about this item? If so, you might compare and contrast it with another recent case we pursued [BMC Public Health LTE Example.pdf (attached)] which was an exemplar situation where the journal was responsive to the ethical issues we raised.

I don’t have any advice on what they should do! The flood of bad research is just so overwhelming. But I can blog it. So I did.

David Allison often works in nutrition science, an area where our government is spreading misinformation that is then sane-washed in the news media. And, along with all that, there’s so much uncertainty and variation that it’s hard to know what to think or do.

Why do we like wind chimes?

Dmitri Tymoczko writes:

You know your question, “why do we like to listen to music over and over again?”

I think the question reveals that our relation to music is sensory and pleasure-based rather than cognitive and information-based. Music (like food and alcohol and sex) gives us pleasure, which is why we do it over and over.

Anyway, I am writing because there is a mirror version of the question: “why do we like wind chimes?” Wind chimes are essentially random notes, but people like them. We don’t really like random words, because our relation to words is information based.

I wanted to title this post, “Why do we like to ask, ‘Why do we like to listen to music over and over again,’ over and over again?”, but I decided that was too cute so I switched to the more SEO-friendly version above. Everybody likes wind chimes, right?

To answer Dmitri’s question, my personal theory of “Why do we like to listen to music over and over again?” is that there’s a big difference between listening to music actively (or playing music), and listening passively (which is how I usually do it).
Continue reading →

Seeking to interview social scientists experimenting with LLM simulations of opinions or behavior

This is Jessica. As part of our interest in how social scientists are using LLM simulations of human behavior, some collaborators and I are seeking interview participants. We are thinking about tools to support rigorous design and evaluation of such studies, and would like to hear examples of goals and challenges that researchers who are experimenting with LLM simulations are facing.

Specifically, we’re looking for researchers in academia or industry who are currently using LLM simulations in their work and could bring a simulation dataset (and optionally, human ground truth data) that they are working with to the interview. We are interested in use cases like piloting surveys or experiments with LLMs, replacing or supplementing human participants in surveys or experiments with LLMs, or using LLM simulations for post-study analysis or theorizing. The interviews should take no more than 1 hr.

Email me and Xudong Tang ([email protected]) if interested, and/or pass this on if you know of social scientists who might be.

How surprising is it that LeBron averaged 27 points, 7 rebounds, and 7 assists in his career but never had one game where he had 27 points, 7 rebounds, and 7 assists?

Joshua Brooks writes:

Listening to a Bill Simmons podcast and the stat came up (not confirmed) that over sixteen years Lebron averaged 27 points, 7 rebounds, 7 assists but never had one game where he had 27 points, 7 rebounds, and 7 assists.

Bill and his guest thought that’s a crazy improbable stat and I guess maybe intuitively it does, but while I’m no statistician I’d imagine the three dimensional nature of the stat would mean it’s not all that in improbable.

But what’s interesting is the category: what is the category of probabilities that seem intuitively (to the average Joe) as improbable but mathematically, are probable? And what are the characteristics of stats that fall into that category?

Hmmm . . . just to do a very crude calculation, suppose that in any game LeBron could score between 17 and 37 points, between 2 and 12 rebounds, and between 2 and 12 assists, with uniform distributions, then his probability of scoring exactly 27/7/7 would be approximately 1/2000. It turns out that he’s played approximately 2000 games, so it wouldn’t be a surprise if he’s ever had a 27/7/7 game but it’s not a surprise if it’s never happened. (I don’t know if it’s really never happened; I’m just relying on a third-hand report here.) I guess we should really bump up the probability a bit here, given that 27, 7, and 7 are in the middle of the distribution, but even if an appropriate probability is 1/1000, it’s still no shocker for it not to happen in 2000 games.

Brooks points to this clip where Simmons describes it as a “crazy” stat, but I agree that it’s not so crazy as all that. Bill Simmons is entertaining and thought provoking, but he’d never be mistaken for a quant.

Frustration when authors dodge criticism

Markus Löcher writes:

I had to think of our discussions on the issue of (not) reproducible research which seems to be a real problem, when I carefully examined the code for this paper, “Stacked regressions and structured variance partitioning for interpretable brain maps,”
published by authors from the machine learning power house CMU.

I quickly discovered a major problem when I noticed that instead of the claimed partial R2 the code computed the marginal R2, which leads to completely different conclusions.

I communicated this to the authors last May and it took them until now finally contact the journal and suggest a small correction, which in my eyes is far from sufficient.

I wrote up my main critique here.

I haven’t read the above paper or critique in detail, so I’m just sharing the story, not trying to make a judgment.

P.S. When following the above link to Löcher’s blog, I noticed this other post about “peeking in AB tests,” so I thought I’d to my post explaining why I think sequential data testing is ok, which is related to my view that type 1 error rate control is not a good thing.

P.P.S. Löcher informs us that the correction has finally been posted.

Why do is it so hard to get a cell phone number with no 0’s or 1’s?

Many years ago—over two decades ago, actually!—I got a cell phone for a family member. I thought it would be fun to have a phone number whose digits spelled out a word (when converting to the 2 = ABC, 3 = DEF, etc., code of the telephone). I had the impression that you could pay the phone company some extra fee and you could pick the number, kind of like a vanity license plate or those businesses that have numbers like CAR-WASH or whatever. It turned out that you can’t do that for a cell phone, or at least this provider didn’t do it. So I said, ok, just give me a number with no 1’s or 0’s, and then I can make a word out of it myself. The person on the line said they couldn’t do that either, but they could give me three numbers and I could pick one. So they gave me three numbers, but all of them had at least one 1 or 0. The person said, no problem, I’ll give you three more numbers. Again, all had at least one 1 or 0. We went through this about five more times and then eventually a number came up with no 1’s or 0’s. So I took it. It didn’t make a great word, but I was able to come up with a reasonable two-word phrase.

I know a lot of people with cell phones, and just about everybody has a number with at least one 1 or 0. I don’t know what’s up with that. There are 7 digits, the first of which can’t be 1 or 0. If the numbers were assigned purely at random, the probability of the number having no 1’s or 0’s would be 0.8^6, which is 26%.

So something’s going on here. Where are all the numbers with no 1’s and 0’s going?

The above question came up in a comment thread. I’m well aware that, back in the old days, there were rules for area codes and phone numbers (all the area codes had 0 or 1 in the middle and none of the three-digit exchanges did), also various reserved numbers such as 411, 911, and 555. But I don’t see any connection between those rules and the current practice where almost all the cell phone numbers have at least one 0 or 1.

Whassup?

OK, here’s a new scam. But I can’t figure out how it’s supposed to work.

The following arrived in my inbox:

Dear Prof. Andrew Gelman

Good day. We are writing with a special academic invitation for Hong Kong this December.

We are an independent academic coordination team based in Japan, and we are helping to coordinate the 2nd Chen Tingzuo Immunological Science Award Ceremony, planned for early December 2026 in Hong Kong. In addition to the award ceremony, the program will include academic lectures and exchanges across different areas of science, medicine and technology. . . .

For this year’s program, we would be very happy to invite you to join us in Hong Kong and give an academic lecture.

The lecture topic is completely open and does not need to be related to immunology. You would be very welcome to speak about your own current research, recent developments in your field, or any subject you feel would be meaningful to share with the audience.

We would be pleased to provide an honorarium of USD 5,000, together with round-trip business-class airfare and local arrangements. A dedicated assistant and interpreter can also be arranged throughout your visit if helpful.

The exact date in early December is still being coordinated. May we kindly ask whether you might have some availability during that period and whether a visit to Hong Kong might possibly work for you?

If there may be some possibility, we would be very happy to share more information and also arrange a short Zoom sometime convenient.

Thank you very much for your time, Professor. We sincerely hope there may be an opportunity to welcome you to Hong Kong.

Warm regards,
**

Obviously this is spam. I have occasionally spoken at medical or public health conferences, but nobody’s gonna fly me to another continent and pay me $5000 to speak on a random topic at such a venue.

The thing that I can’t figure out is . . . how’s the scam supposed to work, exactly? Am I supposed to reply and then they’ll tell me that the free flight and the $5000 check are coming, but first I need to send them some up-front money? That’s too obvious, right?

I’m almost completely sure that the bot that sent the email has no connection to immunology, given that it comes from a weird goacadco.info email address. So, sure, they’re either making up a conference or spoofing a real conference. But, again, to what end? I respond and then they tell me that, by the way, they have a wonderful business opportunity to offer me?

In comparison, the Dr. Ella scam makes a lot more sense to me: I can better envision the sequence of events by which the spammer could eventually profit. This immunology thing . . . I just don’t get it.

The head and the heart

Jackson Lears writes:

Characterising Carter’s views on the nuclear arms race, Luce resorts to a familiar – and misleading – antithesis: the president’s ‘heart’ hated nuclear weapons, while his ‘head’ feared vulnerability to a Soviet first strike. One could just as easily say that the effort to move towards nuclear disarmament came from the head, as the only sane alternative to the fantasy of a winnable nuclear war, and that the fear of a Soviet first strike came from the heart – from the terror that the Soviets were so maddened by ideology as to provoke their own annihilation by America’s weapons.

Good point! I think this problem often arises with head vs. heart arguments, that by labeling one position as the “head,” you’re already rhetorically stacking the deck.

This is kinda related to the One-Way Street Fallacy: Considering only one possibility of a change that can go in either direction. In both cases, the framing of the problem is implicitly doing most of the work.

Overall I disliked Lears’s article—it seems a bit “tendentious,” as the scholars like to say. For example, he writes that CIA backing of jihadist groups “provoked the Soviet invasion of Afghanistan in 1979,” quoting Zbigniew Brzezinski as saying, “We did not push the Russians into intervening, but we did knowingly increase the chances they would do so.” The quote is interesting but I don’t buy the “provoked” thing: Lears is committing the attribution error of assuming that “we” have agency and “they” don’t. Nothing was forcing the Soviet Union to invade a neighboring country. This is not to say that the CIA behavior was appropriate or wise, just that I think Lears is misattributing the blame.

I guess that Lears has a thing about Russia, because later in his article he refers to Putin actions toward Ukraine as “Russia’s concern to maintain friendly relations with its ‘near abroad.'” Friendly relations, huh? Nothing shows friendship like invading and bombing the hell out of a country, I guess. With friends like these . . . . He also writes that Putin “focus on his own nation’s interests,” but I don’t actually see why it’s in his nation’s interests to start an aggressive war. I wonder whether Lears thinks the U.S. government is focusing on our nation’s interests with its bombing campaigns.

Anyway, no need for you to agree with me on the details. The reason for my post is to share my appreciation for Lears’s point about head-and-heart arguments. I put in the rest just to note that I have my problems with his whole article. That’s fine; someone can make a good point within a larger argument that I disagree with.

JAMA rules of engagement on causal and predictive language

Erik, Witold, and I recently published a paper in the Journal of the American Medical Association. This was cool. A day after the paper came out, I ran into someone I know on the street and he said he liked our article. That’s kind of amazing, as usually it seems that nobody reads anything. We’ll have to publish in JAMA more often.

As part of the publication process, we were sent some general reporting requirements per JAMA standards. Here’s the part that will interest you:

USE OF CAUSAL LANGUAGE
Use of causal language (including use of terms such as “effect,” “efficacy,” “cause,” and “x increased y”) should be used only for randomized trials. For all other study designs (including meta-analyses of randomized trials), methods and results should be described in terms of association or correlation (eg, “x associated with an increase in y”) and should avoid cause-and-effect wording.

Avoid use of the words “predict” and “predictive of” for reports of observational studies assessing independent and dependent variables, unless your study design is a diagnostic/prognostic study and has used an appropriate approach to quantifying predictive performance such as (but not limited to) split sampling, cross-validation, or bootstrapping.

On the first item . . . Wow. I wonder if they’d let you write, “Cigarette smoking causes cancer in humans”?

I guess the second item is ok if it’s interpreted broadly enough. For example, Bayesian methods use an appropriate approach to quantifying predictive performance, so I guess that would be included in the “not limited to” category.

Columbia University says, “We believe in difficult conversations at Columbia—in a broad embrace of freedom of expression. It’s in our DNA.”

Claire Shipman, the president of Columbia University, writes:

We believe in difficult conversations at Columbia—in a broad embrace of freedom of expression. It’s in our DNA. I intend to uphold that tradition for several reasons.

First, as a nation, we should be able to tolerate ideas, facts, and opinions that we don’t like. Tolerance is essential to building resilient individuals and a resilient democracy.

Second, it’s our mission. We have a responsibility to guide students toward intellectual rigor, to enable them to grapple with, and examine carefully, ideas different from their own. More broadly, a willingness to test assumptions is essential to our academic excellence in every field.

Third, and perhaps most important, universities—Columbia in particular—are uniquely positioned to help change the national tone. We can guide and model the art of examining complex, potent issues in a deliberate, respectful fashion. . . .

Our instinct might be, in such an era, to shy away from challenging events and speakers. The better course is to model how to do it well . . . Our statutes lay out quite clearly and sensibly that expression, on our campuses, does have limits, in particular as regards harassment, discrimination, or the threat of violence. Peaceful protest, civil discourse, a respectful exchange of ideas—this is our inheritance at Columbia, and I want to ensure that it is also our legacy. . . .

So, as this week begins, and as we continue to debate the affairs of the world and this nation on our campuses, consider this: at times there will be speech that each of us finds disagreeable, even abhorrent—and we still need to protect it. Maybe even seek it out. Such a practice makes us stronger.

I agree. And, although our president did not say so in this message, this policy of protecting and welcoming dissent, even when it is controversial or disagreeable, runs counter to some actions of our university in recent years:

  • The delayed reaction to the sexually-assaulting doctor in the medical school, without any seeming consequences to the people at the university who enabled this behavior.
  • The fake U.S. News numbers, where, again, there seems to have been very little interest by the university administration in finding out who did it and why. As with the medical school scandal, the avoidance of any serious investigation does not seem consistent with the goal of promoting free speech.
  • Barnard College, a division of Columbia University, attempting to cancel the showing of a controversial film on campus. At that time, it seemed that everything was about avoiding offending anybody and about avoiding any controversy—really the opposite of the view that “we should be able to tolerate ideas, facts, and opinions that we don’t like.”
  • Barnard College “mandating that students remove any items affixed to room or suite doors” under the rationale that they “do what we can to support and foster the respect, empathy and kindness that must guide all of our behavior on campus.” Empathy and kindness are fine, but, again, it contradicts the idea that “there will be speech that each of us finds disagreeable, even abhorrent—and we still need to protect it.”

As I noted the other day, Barnard College has also announced a new policy promoting free expression, and I think it’s a good thing.

I completely agree with the president of Columbia that “we should be able to tolerate ideas, facts, and opinions that we don’t like,” and I completely agree with the president of Barnard that “Colleges and universities need to reconfirm our commitment to nonviolent forms of disagreement — even when we are confronted with voices that disparage or dismiss identities and worldviews. This is also a time to foster more disagreement, not less.” I was getting very frustrated with all the happy talk and the official attitude that we shouldn’t be disagreeing. The university—and, of course, the country—has people who disagree with me on important issues, and people I don’t respect at all, for moral, intellectual, or political reasons. And I guess lots of those people don’t respect me either! Which is too bad, but we still have to exist, and we should have the right to express our views, both publicly and privately. We shouldn’t be fired for our nonviolent acts (except to the extent that they interfere with our ability to do our jobs), nor should be discouraged by the university from expressing them.

And, just to be clear, no one at Columbia has ever suggested or even requested that I keep my mouth shut in class or outside of class, including when I’ve criticized the government or the Columbia administration or anything else. I have nothing to complain about on my end, but I do think it was a mistake for them to crack down on dorm room doors, to try to stop the showing of controversial movies, and to keep quiet the bad behavior regarding the fake statistics and the terrible actions at the medical school.

I’m happy about this new policy and I hope the administration follows up by changing the dorm room door policy, apologizing for trying to suppress the movie, and starting and publicizing independent investigations into the U.S. News numbers and the enabling of the sexual assault at the hospital.

Also let me emphasize that by bringing up these earlier cases, I’m not criticizing the Columbia president Shipman, as these things were done before she took office. I also understand the desire to move forward rather than looking back; still, I think that a great way to reinforce the policy of enabling difficult conversations is to revisit some recent examples where the university had unfortunately avoided doing so.

“It’s in our DNA.”

One thing that jumped out at me when reading the president’s email was the statement that freedom of expression, especially in difficult conversations, is “in our DNA.”

My first reaction was the annoyance of the persnickety social scientist: calling it “DNA” is just wrong, and it indeed muddies the waters, as we’re talking about something social, not genetic.

But that reaction of mine was silly. Obviously no one when using “DNA” in that sense is taking it literally; it was a metaphor. And maybe we can gain some insight by taking this metaphor seriously.

For something to be in an organization’s “DNA,” that would imply that it’s not just a transient trait; rather, it has to be in some equivalent of self-replicating code. The idea is that freedom of expression and willingness to confront uncomfortable ideas are not just attitudes that sometimes has existed at Columbia, but rather that they are embedded deeply enough that they will persist. Indeed even if free inquiry and expression at Columbia are suppressed (as with the sexual assault, the fake statistics, the dorm-room censorship, and the attempt to suppress the film screening), the self-replicating aspects of our university’s rules and culture will cause them to return.

And, from that perspective, I do see something in our president’s analogy to DNA. Columbia has for many years celebrated the legacy of the 1968 protests, while recognizing that the other side at the time had legitimate concerns too. That is, Columbia has recognized the value of protest without implying that any particular protest action has to be recognized as being a good idea. We have the University Senate, which dates from that period, as well as generations of new students and staff who have exercised the freedom of dissent—even dissent that some can find disagreeable—and recognize this as a Columbia tradition.

Indeed, this recent statement by the president can be viewed as a consequence of that DNA, an expectation that our university is a place with diverse enough attitudes that there will be people who disagree with and even despise each other but still have the right and, at times, the obligation, to air these uncomfortable disagreements publicly.

Because the alternative, of simulating a nonexistent harmony, would be intellectually dishonest and politically dangerous.

The problem is noise, not N: why increasing the sample size isn’t all that.

Someone was talking with me about the problem with lots of published social science research that’s based on small sample sizes. And I agree that this is a problem! But I think the real problem is noise, not N.

Noise (variation in effect size, variation in outcomes, and measurement error) is more of an issue than sample size because lots of bad studies are so noisy that, to get a large enough sample size to detect a reliable effect, you’d need N = a zillion, and at that point you’d be averaging over such a wide range of conditions that it’s not clear what you’re estimating anyway.

There’s a widespread misunderstanding by which people think that their problems can be solved with bigger samples. I think the erroneous reasoning goes like this: People are aiming for 80% power, but they really have only 50% power, or 30% power, or something like that. If you have 50% power and double your sample size, you’ll have 80% power . . . a win! The error in this reasoning is that often the power is much less than 50%, more like 5% + epsilon. The flip side of this is that if you really are running studies with 50% power or even 30% power, you’re likely to get the right direction on the effect size anyway, so often there’s no good reason to wait on statistical significance.

So, lots of moving parts here: There’s the underlying effect size, the variation in effect size, the variation in the measured outcomes, and researchers’ expectations when setting effect sizes and interpreting results.

Digital twins version 2.0: The skeptics speak up

This is Jessica. Several years since the earliest research papers on the topic came out, the idea of using LLMs to simulate human behavior like survey responses or experiment outcomes (“silicon sampling”) is hitting mainstream. People tend to bring this up in conversation with me, probably because I’ve written about validation and taught a course, so I get to hear a lot of other researchers’ views. I would put the majority of takes I encounter in either the more agnostic, “Somebody should probably explore how much we can do with them” and/or “I’m pretty doubtful it’s going to live up to the hype” camps.  

One way to use silicon samples is digital twinning, where simulations are conditioned on considerable information about specific individuals, and expected to predict the responses of those specific individuals. Early papers on digital twins sing their praises, showing for example that they can reproduce people’s responses more than 80% as well as people replicate their own responses (e.g. here or here). But we’re now seeing a more critical empirical push, which uses novel tasks and topics and more careful design of metrics and baselines for comparison to gauge how much conditioning on more individual-specific information improves prediction. 

In the last month, there have been a few noteworthy data points. Pew Research put out a report describing what they learned from comparing digital twins implemented by giving Claude Opus 4.6 detailed 2025 survey responses and demographic persona information for people who responded to several more recent survey waves. They compute the average absolute error in percentage points for each multiple choice survey question (i.e.,the average of the absolute differences in the proportion of LLM versus human respondents selecting each answer option), and find it averages 12 percentage points over the 300 survey questions they looked at, with 28% of the questions having error over 15 percentage points. They observed several kinds of biases that have come up in other research comparisons: less variation in LLM responses compared to human (with some answer options never being picked by the LLM), stereotyping (like predicting that 97% of Hispanic adults are at least somewhat likely to follow the World Cup; in reality its only 43%), overestimating human accuracy on factual questions, and mispredicting sentiment on current events like data centers and face coverings on ICE officers. They stress how hard it is to predict how far off the LLMs will be on a given survey question.  

This list of biases closely resembles those proposed by Tianyi Peng et al. in one of the most comprehensive evaluations of digital twins to date, recently published in Science Advances: Digital Twins as Funhouse Mirrors: Five Systematic Distortions. They collect a new set of 19 preregistered studies involving 164 outcomes, and find that overall, there is only weak improvement over base models and relatively low correlations with human behavior (on the order of 0.2). 

I like this paper a lot, in particular the comparisons they do to simpler demographic prompting and base models. The intuition that has been sold by researchers and companies in the space alike is that with rich enough information about individual people, we can predict their behavior with high accuracy. To make the twins approach worthwhile, we want to see that twinning specific individuals leads to responses that predict variation between individuals better than the demographic baseline. However, some of the early papers advocating for digital twins evaluated them by looking at measures where a model can do well simply by being a good predictor of the average response. For example, you can look at the correlation between each individual’s real and twin response over a set of likert-style survey questions, but a model that does a good job of predicting on average which questions will get higher responses may appear to do well by this measure, even if it does a poor job of sorting individuals for any particular question. 

Instead, Peng et al. look at correlations between human and twin responses for a single outcome at a time, and compare to demographic personas only as well as base models, to ask how much additional predictive power the twin approach provides when it comes to sorting people into those that will respond with higher versus lower values. They find that while there’s a boost in average correlations (from 0.08 with base models to 0.15 with demographic personas to 0.2 with twins), overall the correlations remain modest, and much more modest than the correlations over questions that other papers have reported. Hopefully it encourages more care in future papers when it comes to what measures actually support the claims being made. I do think behavioral researchers and others interested in this space are gradually changing their mental models to conclude that LLMs may do a decent job of predicting population averages for well-represented populations, but they have a long way to go when it comes to modeling individual heterogeneity. There are several other papers along these lines that I haven’t had a chance to look closely at. 

How damning is this for the approach itself? I think it remains hard to say, though it certainly suggests skepticism is warranted at this point in time. The reasons I wouldn’t write digital twins off completely are 1) it is still early and models continue to improve and 2) prompting is often not the best way to recover the predictive information a model has–you can do better at opinion prediction by using model internal representations (see, e.g., this). On the other hand, ultimately it comes down to limits on how predictable individual behavior is, and while people are very predictable in lots of ways, we are also always exercising our agency and changing in subtle ways that interact with the world, which even we ourselves can’t necessarily understand or predict. So I don’t think reaching AGI is synonymous with being able to predict the majority of individual opinions and actions. I’m reminded of Matt Salganik et al’s work on the Fragile Families Challenge, predicting life outcomes from thousands of variables, where the best of the 160 teams’ models explained only about 20% of the variation on the most predictable outcomes, despite the enormous amount of data. 

What I will say with certainty is that silicon sampling has attracted some very good salespeople. I expect a lot of money will be made before anyone abandons the idea.