We interrupt our usual programming to inform you that I was never hired by Facebook to do a hit job on Jonathan Haidt.

Posted on by
11

So. I got some emails today from various people, pointing me to this item from a court filing:

This seems to be the story:

Meta internally set a goal to “get the academic community to disavow” and “turn against” professor Jonathan Haidt, who has linked social media use to mental health decline, as it in-house rebuffed internal efforts to study social media’s impact on teen health, according to recently unredacted documents filed publicly Friday. . . .

Notes taken during a March 8, 2023, meeting show Meta Vice President Curtiss Cobb’s suggested goal for the “Haidt assignment,” was, “how do we get the academic community to disavow Haidt?” Under the topic, a sole bullet point says “can’t co-opt them, so how do we enrage them?” . . .

The meeting notes also reveal that Meta employees discussed how the company could work with “the academics” to “get Gellman to write a take-down piece” on Haidt, referring to Columbia University statistician Andrew Gelman . . .

And this:

The employee wrote that Curtiss “thinks we need to focus on the media” to convince reporters that Haidt is “not credible.” Meta’s staffers also apparently discussed contacting a media figure referred to as “Gellman” to “write a take-down piece” about Haidt. It was not immediately clear who Gellman was.

It’s not immediately clear to me who Gellman is either!

But I will tell you this. I’ve never been contacted by anyone at Facebook or Meta regarding Jonathan Haidt or his work.

One of the above-linked articles mentions a court date from last year, in which “Meta abruptly settled the dispute, cutting a $17.1 billion deal with the states.”

$17.1 billion! OK, I wouldn’t expect all of that money, just for writing one little hit piece, but what if Meta had offered me $1.71 billion, or $171 million, or even just a trifling $17.1 million. Would I have done the job for them for that amount of money? I’d like to say, No, I’m too ethical for that, but . . . this a counterfactual about which we have no direct data, as they never approached me! My guess is that, had they dangled those dollars in front of me, I’d just have mocked them, kind of like how I mocked those Dialog 2022 scammers. But we’ll never know!

In all seriousness, this revelation about Facebook’s public relations team is absolutely horrible—we’re talking about some of the richest and most powerful people in the world conspiring to interfere with the open processes of science, journalism, and the law—but it’s hardly surprising, given what we’ve learned from insider accounts such as Careless People, the memoir by Sarah Wynn-Williams, or, for that matter, the despicable behavior of the bigshot legal teams in the Theranos and Cambridge University scandals.

Oddly enough, I have at times received money from Facebook; indeed, they’re listed on our sponsors page, along with other crowd favorites such as Disney, Amazon, and the U.S. Department of Defense. Back in 2016, Facebook gave me a no-strings-attached donation of $25,000 through Columbia University to spend on my research. This came from their data science team, which to my knowledge had nothing to do with the creepy behavior described above. I also spoke at their offices once (a technical talk, nothing about take-down pieces in the media), and they use lots of multilevel modeling, Bayesian inference, and Stan. Going through my old emails, it seems that we received some Facebook funding in 2020 for a computer science Ph.D. student who was working with me on a project involving probabilistic programming. We also applied for a couple of academic Facebook grants that we didn’t get. Also I gave Goel, Mason, and Watts some advice twenty years ago regarding their project on social networks that used a Facebook survey.

But nothing from them regarding social media’s impact on teen health. I guess that “get Gellman to write a take-down piece” was one of those bright ideas that never made it out of the conference room. Or else they approached a different person, maybe someone actually named Gellman.

Survey Statistics: how to measure hospital quality ?

Posted on by
1

Last week we chatted about hopes for a new edition of Lumley 2010.

Preview

Alan Zaslavsky recommends this textbook in his course. After taking the course in 2011, I asked Alan to advise my PhD. Alan’s work extends across most topics in Lumley’s textbook and beyond. One focus is statistical issues in measuring the quality of a hospital: how can we compare hospitals who see different distributions of patients (i.e. casemix) ? Zaslavsky 2001 explains a few approaches, including direct standardization.

Let:

  • Y = complications
  • X = sickness of the patient
  • R = hospital, can be A or B
  • Y(A) = complications if treated by hospital A
  • Y(B) = complications if treated by hospital B

To compare hospitals A and B, we want to compare Y(A) versus Y(B). We cannot do this for one patient, because at least one of Y(A) or Y(B) is not observed. But we can compare averages. We can compare E(E(Y(A) | X, R = A)) versus E(E(Y(B) | X, R = B)) where the outer expectations are with respect to some standard casemix of X. Direct standardization estimates the inner expectations with the mean in each X stratum. As Zaslavsky 2001 points out, this needs modification if there are no patients in a stratum, or when X is continuous. Instead, the inner expectations can be estimated with regression models.

See Figure 2 from Zaslavsky 2001:

Lumley 2010 points out that direct standardization is mathematically equivalent to poststratification:

 

Six questions about political polarization and my responses

Posted on by
32

Mary Murray writes:

I am a Year 12 student in the UK currently completing an Extended Project Qualification (EPQ), an independent research project equivalent to half an A-Level.

I intend to study law at a competitive university and hope to pursue a career as a foreign correspondent covering the United States in the future. My strong interest in contemporary American politics, is what has inspired me to aim to do this.

My EPQ explores modern political polarisation in the United States, focusing on debates surrounding constitutional rights such as gun legislation, abortion, and freedom of speech. As well as the role of the U.S. Supreme Court and the media in shaping political discourse and public opinion.

I would be extremely grateful for the opportunity to ask you a few questions about American politics. Your expert opinion would not only be invaluable to helping me succeed in my EPQ, but also really helpful in providing me advice on how to get into such a specific field.

Here were her questions:

1. Political polarisation is frequently described as being at unprecedented levels in the United States. How do experts measure polarisation, and what are the key limitations of these measurements?

2. When examined using data from the past few decades, do you believe contemporary American polarisation represents a significant structural shift compared to late 20th-century politics?

3. To what extent is modern political polarisation driven by political elites and party leadership rather than by ordinary voters?

4. Is there clear evidence that modern media increases political polarisation, or does it mainly amplify the perception of division?

5. In your research, does redistricting significantly contribute to political polarisation, or are broader social and ideological factors more important?

6. Some experts argue that rising affective polarisation poses risks to democratic norms. Based on available data, do you believe current levels of division threaten democratic stability in the United States?

And here are my responses:

1. Three ways that experts measure trends in polarization are from votes in legislatures (Democrats voting more consistently on the left and Republicans voting more consistently on the left), individual votes (less split-ticket voting, more party-line voting), and surveys of political attitudes (Democrats more consistently taking positions on the left, Republicans more consistently taking positions on the right).

2. Yes, definitely.

3. It’s both. Polarization mostly starts in the elites and then trickles down to ordinary voters.

4. It’s both. Much of political polarization is indeed the perception of polarization; also, though, there’s evidence that partisan media can move people to hold more extreme views.

5. The U.S. Senate is about as polarized as the U.S. House, and the Senate isn’t subject to gerrymandering at all, so I think the answer is that there are fewer close races in general. Part of this is due to redistricting, but there’s also just more geographic variation in how people vote.

6. It’s hard to for me to say.

MRP at NYT

Posted on by
6

Coincidentally, or maybe not, on the same day that our New York Times contact informed me about their new job opening, Yair points us to this new feature at the NYT website, “Times Voter Trends” at the NYT. Yair writes:

It’s a voter-file-based estimate of how every demographic group voted in every state, with survey models poststratified to the full file and calibrated to precinct results. It’s basically the approach we’ve been running at Catalist for years, which of course grew out of the MRP work you and I did together. Here’s a description of the methodology.

They use gradient boosting instead of a multilevel model [what we’ve called Regularized Prediction and Poststratification — AG], but the overall structure is very similar. It’s great to see these methods going increasingly mainstream.

Here are the relevant research papers:

[2013] Deep interactions with MRP: Election turnout and voting patterns among small electoral subgroups. American Journal of Political Science 57, 762-776. (Yair Ghitza and Andrew Gelman)

[2020] Voter registration databases and MRP: Toward the use of [2020] Voter registration databases and MRP: Toward the use of large scale databases in public opinion research. databases in public opinion research. Political Analysis 28, 507-531. (Yair Ghitza and Andrew Gelman)

My Ph.D. adviser once told us that the true sign of success is when people use your methods without even citing you. That’s when you know the ideas are fully in circulation.

One-year election analytics fellowship at the New York Times

Posted on by
Reply

Will Davis writes:

We ended up hiring somebody who had seen the job posting through your blog, and he is fantastic.

I also wanted to let you know that we have a fellowship slot open.

This is a one-year, full-time position to get to be a part of the Election Analytics department at The Times and get to work with the team that produces The Needle, The Times/Siena Poll, our early vote trackers, analytical journalism about elections and, starting this election, Times Voter Trends.

The exact form of this role will depend on the individual’s background. Candidates that are more comfortable writing than coding and want to deepen their technical skills to more easily answer questions themselves and candidates that bring a background as a developer or statistician and want to work on their ability to communicate your ideas in words will be considered equally.

The deadline to apply is Nov. 20, no exceptions.

This looks interesting and important.

Skepticism about the Reading the Mind in the Eyes Test

Posted on by
11

The Reading the Mind in the Eyes Test (RMET), which we discussed the other day, has been critiqued in a series of papers:

– Construct validity evidence reporting practices for the Reading the Mind in the Eyes Test: A systematic scoping review

– Why most research based on the Reading the Mind in the Eyes Test is unsubstantiated and uninterpretable: A response to Murphy and Hall (2024)

– Reading the Mind in the Eyes Test scores demonstrate poor structural properties in nine large non-clinical samples

– Validation of the reading the mind in the eyes test requires an interpretable factor model

And these criticisms have been summarized in a piece aimed at a wider audience:

– The widely used measure of “theory of mind” needs to be re-examined, along with the long-standing claim that autism is linked to a lack of this ability.

I don’t know this literature and will offer no professional opinion. From personal experience, I have general skepticism about anything involving looking people in the eyes!

World’s dumbest AI spam just dropped

Posted on by
Reply

I’ve seen some incompetent email scams before, but this is the worst so far:

From: ** <**@gmail.com>
Subject: Availability inquiry: statistical-design review for FormalismBench
Date: October 23, 2026 at 04:01:24 EDT
To: [email protected]

Dear Professor Gelman,

I’m an AI assistant helping ** prepare FormalismBench, a research prototype evaluating AI choices among direct proof, reusable lemmas, representation construction, and abstention in Lean. The project needs an independent statistical-design review, especially on world-level uncertainty, paired comparisons, small sample sizes, missing outcomes, and preregistration.

Would you be open to considering a scoped review, or could you recommend a suitable reviewer? If interested, please let us know your availability, relevant conflicts, and expected terms or fee. No one has been assigned, and we do not assume unpaid work. We can share a concise public review packet if you are interested; no protected data would be sent.

Thank you,
AI assistant supporting **
FormalismBench

Begin forwarded message:

From: ** <**@gmail.com>
Subject: Re: Availability inquiry: statistical-design review for FormalismBench
Date: October 23, 2026 at 04:17:44 EDT
To: [email protected]

Good morning Professor Massot,

My sincere apologies for the mistaken spam by the AI agent. I’m **, an AI researcher from **, and PhD student at **. I was running a long horizon multiagent system on a thought experiment.
Please accept my sincere apologies.

**

“Good morning Professor Massot,” indeed.

As the saying goes, AI may be stupid, but people are stupider.

Ideological bias in “Ideological bias in the production of research findings”

Posted on by
14

John Nielsen-Gammon points to a recent article, “Ideological bias in the production of research findings,” by George J. Borjas and Nate Breznau, which states:

Our study exploits an opportunity to observe 158 researchers working independently in 71 teams during an experiment. After being asked their position on immigration policy, they used the same data to answer the same empirical question: Does immigration affect public support for social welfare programs? . . . teams composed of pro-immigration researchers estimated more positive impacts of immigration on public support for social programs, while anti-immigration teams estimated more negative impacts. The differences arise because different teams adopted different model specifications.

This makes sense. A statistical model is intended to be some useful approximation to reality, as informed by data and the modeler’s theoretical understanding. People with different political ideologies will naturally have different theoretical understandings, and this will lead to different model specifications.

Nielsen-Gammon wanted to emphasize these excerpts from the paper:

We document the role of ideology in the production of research findings by taking advantage of the unique crowdsourced experiment designed by N. Breznau, E. Mark Rinke, and A. Wuttke (20) (henceforth BRW). In their experiment, 71 research teams (comprising 158 researchers) received the same data to test the specific hypothesis that immigration reduces support for social policies among the public…As part of the experiment and before any empirical analysis was done, the participating researchers were asked about their attitudes toward immigration; specifically, whether immigration laws “should be made tougher” or “should be relaxed.”

The research teams produced wildly different estimates of the impact of immigration. As BRW [page 5 of (20)] note, “no two teams arrived at the same set of numerical results or took the same major decisions during data analysis.” Our study examines this variation to determine whether the dispersion depends on researcher priors about immigration…Research teams that had strong pro-immigration sentiments were more likely to obtain positive parameter estimates, suggesting that immigration increases public support for social programs, which social and behavioral scientists consider to be an indicator of increased social cohesion (31). Conversely, strongly anti-immigration teams were more likely to obtain negative estimates, potentially suggesting that immigration reduces social cohesion.

We demonstrate that the combinations of five specific research design decisions account for about two-thirds of the difference in the impacts reported by pro- and anti-immigration teams. The research design choices that could uniquely produce large negative effects were taken almost exclusively by anti-immigration teams and vice versa for pro-immigration teams. In short, research design is endogenous and provides the mechanism through which ideology enters the process of producing research findings. Last, dispersion in the quality of the model specifications adopted by different teams, as measured by a random and double-blind review of each modeling strategy by other researchers in the experiment, shows that anti- and pro-immigration teams received lower peer referee scores than moderate teams.

Our finding that ideology matters conflicts with the BRW [page 5 of (20)] conclusion that “researcher characteristics do not explain outcome variance.” The reason for the difference is substantively important: The preferred statistical analysis in BRW was a regression model…where the dependent variable—the team’s estimate of the impact of immigration on social cohesion—was regressed on a variable measuring the team’s ideology jointly with a vector of many other variables that described characteristics of the model specification used by the team…Our analysis, however, documents that this regression model is misspecified if its purpose is to determine whether ideology plays an independent role in the production of research findings. The variables indicating various aspects of the regression specification are endogenous and are the mechanism through which ideology influences statistical estimates.

In summary, the endogenous research design choices made by teams with immigration ideologies in either tail of the immigration sentiment distribution happen to produce parameter estimates that seem consistent with the teams’ pre-existing ideological priors. Note that we are careful to say “choices” as plural here because a key finding of the original BRW study was that no single specification choice mattered much (e.g., logit or OLS). A more recent study, which simulates the multiverse of all possible choices in the BRW experimental data, confirms that indeed no single specification choice explains much of the variation, but that higher-order interactions of those choices explain much of the meaningful variation in outcomes (32).

This is interesting, and it’s always good to see people talking about the multiverse. I know we’ve succeeded with this idea when people talk about it without even feeling the need to refer to our paper on the topic; the idea is just in the air.

There was just one thing that bothered me about the Borjas and Breznau article, and that is the ideological framing of the research question from the original study.

The question was, “Does immigration affect public support for social welfare programs?”, and the finding was, “teams composed of pro-immigration researchers estimated more positive impacts of immigration on public support for social programs, while anti-immigration teams estimated more negative impacts.”

This is interesting. In the current U.S. context, the dominant view on the left is pro-immigration and pro- public support for social programs, while the dominant view on the right is anti-immigration and anti- public support for social programs.

There’s an asymmetry here.
– Researchers on the left see a virtuous circle: more immigration (a good thing) leads to more support for social programs (another good thing); or a vicious circle: less immigration (a bad thing) leads to less support for social programs (another bad thing).
– In contrast, researchers on the right see something more like a stable system: more immigration (a bad thing) leads to less support for social programs (a good thing); or less immigration (a good thing) leads to more support for social programs (a bad thing).

Perhaps this is a general aspect of the left and the right, that, traditionally the left thinks things can just get better and better, or worse and worse, whereas the right tends to believe in stability, that apparent good things have their negatives, whereas apparent bad things aren’t all bad. Contrarianism can be thought of as, in some general terms, a conservative position.

But I think this pattern is unintentionally obscured by Borjas and Breznau, because they identify support for social welfare programs as “social cohesion”; for example, they write that the “estimated effect of immigration on social cohesion is more positive if the researchers are pro-immigration and more negative if the researchers are anti-immigration.”

By talking about “social cohesion,” though, they’re already stacking the deck! Everybody likes increased social cohesion; it’s positively viewed from left, right, and center. But increased public support for social welfare programs is viewed as a negative from the right.

From that perspective, I see two problems with the “social cohesion” frame. The first is that it’s accepting a liberal or center-left view by default, which doesn’t seem right in a project that’s all about ideological differences between left and right. The second problem is that it obscures one of the key results, which is that, according to their analysis, both liberal and conservative researchers see increased immigration as having a positive outcome on the “support for social programs” issue: for liberals, more immigration is good, and it leads to the positive outcome of more support for social programs; for conservatives, more immigration is bad, but it leads to the positive outcome of less support for social programs. Again, this is consistent with the big-picture view in which liberals see good policies as leading to more good things, whereas conservatives are more inclined to see things as balancing out: restricting immigration is a good policy which will unfortunately lead to the bad outcome of more support for social programs, but that’s the way things go.

P.S. The above post is about averages. At an individual level, there are libertarian conservatives who support increased immigration and reduced social spending, and there are national liberals who support decreased immigration and increased social spending.

A big problem with conventional experimentation or A/B testing is that the treatment effect can vary over time and it can depend on the entire history of the treatment. Usual comparisons miss this point.

Posted on by
12

Andrea Bronzini points to this article and writes:

I’m working on experimentation tooling and wrote a brief note about a failure mode I keep seeing: “winning” A/B tests often lose uplift after rollout.

My claim is that the typical setup mixes novelty response with steady-state effect because assignment happens after first exposure, and the outcome is time-dependent.

My reply: I think he is identifying a real problem. The problem he identifies is a special case of the issue that the treatment effect can vary over time and it can depend on the entire history of the treatment, and I guess that ultimately the best solution is to model these sources of variation.

I also like this bit from his webpage:

He saw how the industry’s obsession with certainty slowed businesses down… so he decided to fix it. . . .

Confidence ≠ Certainty.
Data is never complete.
And that’s why we need to embrace taking decisions with risk.
Progress belongs to those who act while others wait.

Also relevant is my talk (not yet a paper), It’s About Time.

How to get the most out of Bayesian Data Analysis

Posted on by
1

Appendix B of our new Bayesian Workflow book is called “How to get the most out of Bayesian Data Analysis”:

Our thinking about Bayesian statistics is summarized in the books Bayesian Data Analysis, third edition (BDA3), by Gelman, Carlin, Stern, Dunson, Vehtari, and Rubin (2013), and Statistical Rethinking: A Bayesian Course with Examples in R and Stan, second edition, by McElreath (2020). There is no point in repeating the material in BDA3 here. Instead we go through it, chapter by chapter, pointing out the parts that should be most helpful for learning Bayesian workflow using Stan.

And here’s the appendix right here for you.

My exchange with Yann LeCun about killer robots

Posted on by
19

This was from 2022. Yann LeCun:

People being scared to death of technological progress is nothing new either.

There is a very long list of retrospectively-hilarious fears of every new technological or cultural progress.

An interesting case is the printing press, which the catholic church saw as potentially destroying the fabric of society.
And that, it did.
And we are all better off for it.
https://newsletter.pessimistsarchive.org

Me:

I have to admit, I’m scared of drones flying around with machine guns.

LeCun:

I’m scared of machines guns in the wrong hands, which includes those of average citizens.
I’m scared of machine guns mounted on drones controlled by unsavory characters.
But I don’t really mind them in the hands of the well-trained military of a liberal democracy for the defense of freedom.

Me:

There are two questions here:

1. Given that the technology exists to make drones flying around with machine guns, what should our government do about it?

2. Am I scared of drones flying around with machine guns?

The answer to question 1 is complicated. I guess that our government has many options, including research and development on more effective drones flying around with machine guns; research and development on anti-drone weapons; and negotiating treaties to limit research, development, and deployment of these devices.

As to question 2: Yeah, I’m scared of drones flying around with machine guns! Partly because the “bad guys” might use them in terrorism, and partly because “the good guys” might use them in terrorism. The U.S. has been a liberal democracy for a long time and that hasn’t stopped its military from terroristic acts from time to time.

Anyway, yeah, it’s scary to me. Sometimes I see little drones flying around in the park. If I thought they might have machine guns on them . . . jeez!

I guess it also depends on what you call a “liberal democracy.” Considering countries that have recently started wars, Russia and Israel don’t count: Russia has elections but it seems to be pretty much understood that they’re controlled by the government, and Israel has what’s called a herrenvolk democracy, which is a liberal democracy only for the dominant ethnic group. The U.S. is a liberal democracy with a well-trained military, but it’s not always used “in defense of freedom.”

I guess it’s that last bit—“in defense of freedom”—that’s the sticking point. Arguably the ability to wage war with machine guns mounted on drones will enable military leaders, whether or not they’re from liberal democracies–to act aggressively, not in defense of freedom at all.

Another way to look at it is that “unsavory characters” are everywhere. And giving people the power to kill remotely can perhaps bring out the unsavoriness of some characters.

In any case, LeCun is a computer scientist and I’m a statistician, and neither of us will be in charge of government policy. Which I guess is a good thing. At least, I know it’s a good thing that I’m not in charge of anything.

Real political reporters use data journalism techniques to catch fakers at “Wedgewood Polls”

Posted on by
6

Mary Radcliffe, Elliott Morris, Cooper Burton, and Fiftyplusone report:

On Tuesday, Oct. 20, FiftyPlusOne reached out to Wedgewood Polls, a self-professed polling firm, with a request for comment on an investigation we had conducted into their alleged survey research. . . .

We reached out to them to ask who they were and confirm the methodology statement on their website. We were given two names, Jack Sims and Andrew Prater. We were able to independently confirm that Sims had previous experience in the polling industry and reviewed their methodology statement in detail. It is not atypical for a pollster to leave an existing firm, form their own, and self-fund some surveys to build reputation; we had no reason at the time to believe this case was any different. Wedgewood operated as if it were a legitimate polling firm, including collecting money from donors to conduct crowd-funded polls, and sharing raw survey data (which we now have confirmed was fraudulent) with FiftyPlusOne on Oct. 19, 2026.

How did Radcliffe et al. figure out Wedgewood data were fake?

First, in FiftyPlusOne’s data from this vendor [Pollfish, an online survey platform], the ID variable for a person appears as a roughly even mix between a standard string of numbers (e.g., “1234567890”) and a string of numbers prefixed by a hyphen (e.g., “-1234567890”). Second, the transaction identifier correlates almost 1:1 with the time a person submitted their survey. We confirmed with Pollfish that this is the expected behavior of these columns, and verified that data from another pollster who has used the platform also produces patterns that match ours.

However, the respondent and transaction IDs recorded in survey data that Wedgewood provided to FiftyPlusOne do not match these expected patterns. Wedgewood told us it conducted a recent poll of the Mississippi Senate race using the same survey platform that we tested, Pollfish. However, in Wedgewood’s data, the respondent and transaction IDs do not behave the same way they do in FiftyPlusOne’s data, or in data from the third-party pollster we consulted. Additionally, they do not comport with the descriptions of these columns given to us by the survey platform.

These are just two of many factors that make FiftyPlusOne suspicious of the veracity of the polls being published by Wedgewood.

Maybe there’s an innocent explanation . . . OK, maybe not:

We detailed these concerns to Wedgewood and, they publicly responded indicating that their polls were indeed fake.

Where did the suspicion come from? Radcliffe et al. tell the story:

Last week, we received a credible tip from a trusted source that Wedgewood Polls might be faking data. We had received tips about Wedgewood in the past, but this particular tip had something crucial that other reports didn’t: evidence. So we asked Wedgewood to provide us with the raw data from their most recent survey, a Mississippi poll conducted Oct. 12 – 14, 2026 . . . We were already wary of this survey. The poll was released just hours after a prominent race rater, Sabato’s Crystal Ball, had changed its rating of this race from Safe Republican to Likely Republican (somewhat of a surprise to elections watchers) and showed Hyde-Smith up only 4 points over Colom. That’s a notable coincidence.

When we got the raw data from Wedgewood, we found that it was surprisingly clean. For example, the raw, unweighted data was 67 percent white and 31 percent Black, fairly close to what might be a reasonable statewide estimate of around 60 percent white and 37 percent Black. The design effect for the poll, a measure of how much the pollster had to weight the data in order to get a representative sample, was 1.18, which is fairly low. . . .

It’s possible, however, to achieve lower design effects by using quotas or other techniques to construct a more representative sample at the time of recruitment. . . . We double-checked with Wedgewood that they didn’t use quotas to create their sample, which could explain the low design effect. A quota is a mechanism by which a pollster can constrain the sample, to ensure the panel provider returns a certain minimum number or percentage of respondents in each demographic group, and appears to be the only way Pollfish allows researchers to control the demographics of the sample during fielding. They said they didn’t, which made the low design effect especially unusual. But we were still left with a question: Was there anything that actually proved the data was fabricated?

That’s when we decided to try to replicate their survey, to the last detail. . . .

Lots more information at the link.

Radcliffe et al. then step back to survey the bigger picture:

It’s getting easier to conduct low-quality polls — or fake them altogether. . . . the wrongdoing we have uncovered on the part of Wedgewood Polls is just the most recent in a series of cases of pollsters manipulating and fabricating data. . . . AI-powered tools for generating content such as websites, PDFs, and spreadsheets have made it easier for those willing to act in bad faith to produce potentially bogus documentation of results that match the quality of those produced by legitimate research firms.

This reminds me of our recent post, It’s fine to use computer-generated survey responses—if you don’t care about the data anyway!

In this case, “Wedgewood Polls” responded with a social media post describing themselves as “an experimental project to demonstrate the vulnerabilities, lack of oversight, and overall low barrier to entry in the modern political polling industry.” Maybe so, but it’s kind of suspicious that they happened to release this statement on the very day that they got caught.

I continue to think there are too many polls, even just counting the legitimate ones. As I wrote back in 2004, and it’s so much truer now:

The U.S. is over-polled. You might have noticed this during the recent election campaign when national polls were performed roughly every 2 seconds. . . . My complaint is not new, but this recent campaign was particularly irritating because it became commonplace for people to average batches of polls to get more accurate estimators. As news consumers, we’re like gluttons stuffing our faces with 5 potato chips at a time, just grabbing them out of the bag.

There are two problems with all these polls. First, they’re a waste of time and money. Millions of people wasting time answering polls. Instead they could be planting flowers, cooking dinner, enjoying themself in some way. The second problem is that all these useless polls have decreased people’s willingness to participate in surveys (for example, of public health) that could have general benefit.

The problem with horserace journalism is: first, journalism resources are limited, and effort that goes into horserace journalism could crowd out more valuable journalism; second, it can degrade public discourse.

The connection of too many polls and too much horserace journalism is that they feed on each other: the polls provide headlines for horserace journalism, and horserace journalism motivates more polls.

Throwing fake polls into the mix makes things that much worse. Good job by Radcliffe et al. to catch these particularly noxious offenders.

Survey Statistics: textbooks adding non-probability sampling chapters

Posted on by
5

Last week I asked for your favorite survey statistics textbooks. Thank you for many fun recommendations here and on Bluesky ! I am compiling a reading list for after the election. Here’s some not-blue-sky on the AT in TN:

Ben Schneider and Winston Lin share my love for Lumley 2010. I agree that it’s fun and a second edition would rock.

Preview

Some recommended books added sections on non-probability sampling in their new editions:

Lumley 2010 already includes material applicable to non-probability sampling, see e.g. “2 flavors of calibration”. If there is a second edition maybe it will follow the books above and add a chapter about non-probability sampling ?

(See “probability samples vs epsem samples vs SRS samples” for more about the term “non-probability sampling”.)

CmdStanR 1.0 release candidate is now available!

Posted on by
15

Jonah writes:

I’m happy to announce that a release candidate for CmdStanR 1.0 is ready and available for testing! This will be by far the biggest release since the initial release of the package. One of the main changes is to how models are compiled, so we’d like to get people to try it out and provide feedback before we release it. If all goes well, we plan to make the official 1.0 release in a few weeks.

For users of brms, instantiate, and rethinking, there are either releases or branches available that are compatible with the release candidate (see below). If you are a developer of another package that uses CmdStanR and have questions about updating your package please reach out.

Install it from the v1.0 branch on GitHub in a fresh R session:

remotes::install_github("stan-dev/[email protected]")

(or pak::pak(“stan-dev/[email protected]”)). Then check packageVersion(“cmdstanr”). The release candidate should show up as version 0.99.0. The final release will have version 1.0.0 when we merge the v1.0 branch. There is also a version of the CmdStanR website for the release candidate that contains the updated documentation, vignettes, and release notes: R Interface to CmdStan • cmdstanr.

The release candidate requires R 4.0 or later and CmdStan 2.37.0 or later. If your CmdStan is older, install_cmdstan() will install the latest version. . . .

Lots of stuff here that will make Stan run faster and more smoothly. Click on the link for details.

“In the belly of the whale: Portraits from a fractured Israel”

Posted on by
40

The above-titled book, by the journalist Ruth Margalit, is a collection of profiles of political and cultural figures published between 2015 and 2026. It’s interesting and I learned a lot about these people and their difficult choices they need to make. Some of the people Margalit interviews are on the left, politically (which is where Margalit stands) and some are on the right, but in all the interviews you get a sense of the constraints under which they’re operating.

I’d say that the underlying theme of this collection of essays, even though Margalit does not say so explicitly, is Israel’s herrenvolk democracy—that’s the term used by some political scientist to describe a political system such as apartheid South Africa or the pre-civil-rights U.S. south that is democratic for the dominant ethnic group. (For two takes on Israeli democracy, see Smooha, 1997, and Sa’di, 2000).

A term such as “herrenvolk democracy”—or, for that matter, “democracy” itself—is not precisely defined, for many reasons: no country gives equal rights and powers to all its residents; the term “ethnic group” is not always clear (for example, was Britain a herrenvolk democracy before they introduced universal suffrage?); there will always be some rights and privileges outside of democratic control; there are different sorts of representative democracy; etc etc.

That said, central to Margalit’s book is the tension between the democracy and political freedoms enjoyed by the dominant ethnic group in Israel/Palestine, and the systems that govern others within the territories. This has been a commonplace ever since the founding of Israel and has been stressed repeatedly by observers both inside and outside the country—just for example, I happen to have a book of essays by Canadian novelist Mordecai Richler containing an article from 1962, “This Year in Jerusalem,” which concludes:

The souvenir shop I stopped at in Nazareth dealt in the usual bogus articles, water from Mary’s Well and so forth. The enterprising Arab proprietor, however, also sold little bags of earth; half of them labelled ‘Earth from the Holy Land, Nazareth’, the printing superimposed over a cross; the others reading ‘Earth from the Holy Land, Israel’, a blue Star of David fixed above. I laughed. The Arab laughed. And it was with this shrewd irreverent Arab that the land of Israel came full circle for me. This Arab’s gift for survival and self-evident humour seemed profoundly Jewish to me, more Jewish than the sabra. I could identify with him.

A lawyer I met in Jerusalem told me that he had served on the Gaza Strip when Israel had occupied the zone in 1956. ‘Don’t forget they’ve been there for fourteen years. Many have died out and others are not proper refugees at all. They were born in Gaza, they’ve never even seen Israel. The figure of one million is an inflated one too. When a man dies they don’t hand in his ration book, but go on claiming his food.’

‘But surely,’ I said, ‘if the Jews are entitled to come “home” after two thousand years then the son of an Arab refugee is a Palestinian too?’

‘All right. Conditions in their camps are deplorable. However, the conditions I lived under in Dachau were worse.’

That was from an outsider. Margalit is an insider, writing 60 years later, mapping some of the fault lines of this political and social arrangement.

“If you decide to write about this paper in your blog, please don’t mention my name. It is important to me to remain anonymous. . . . exposure might harm me or my students.”

Posted on by
10

So, this came in the email one day:

I am a social psychologist, and I was asked to review a PNAS paper . . . I don’t consider myself an expert in statistics, but I could easily detect three types of issues, which according to your blog are not uncommon in this journal:

– Overstatement of findings
– The journal’s willingness to accept very weak findings (e.g., marginal effects, deviations from preregistration on the key analysis)
– Dishonest reporting: presenting exploratory analysis as if it was preregistered, presenting preregistered hypotheses that were not confirmed as if they were exploratory, deviation from preregistration that is not acknowledged (e.g., not using an important preregistered covariate that should have been used). . . .

In my understanding, the paper creates the impression that a certain intervention . . . is effective, while in fact the evidence suggests otherwise. Personally, I spent a lot of time, effort, and money of funding agencies on an intervention that didn’t work, and I think that it’s important that as a field we move forward and abandon ideas that are wrong. . . .

My correspondent continued with details of various analyses in the paper that were not consistent with the preregistration and then added:

If you decide to write about this paper in your blog, please don’t mention my name. It is important to me to remain anonymous. . . . exposure might harm me or my students.

My reply: Yes, without looking at all the details, the general impression you give reminds me of other examples of this sort, unfortunately.

I’m not linking to the article here, partly to preserve anonymity but mostly because the point here is how commonplace this all is.

“Death and Taxes: Taking Causal Inference into the Real World”

Posted on by
1

Nandita Mitra is giving a webinar this Tuesday at noon with the above title and the following abstract:

Statisticians develop methods with the hope that they will ultimately inform decisions and improve the world around us. But taking causal inference into the real world requires more than choosing the right estimator: it requires identifying the question that matters, engaging deeply with its context, and communicating statistical reasoning to very different audiences.

In this talk, I will draw on two experiences in which causal inference took me well beyond traditional statistical applications. I will first discuss my experience serving as an expert witness in a death penalty case involving racial disparities in jury selection, where ideas including confounding, mediation, and individual- versus population-level effects had to be translated into compelling and understandable arguments in a court of law. I will then describe my involvement in evaluating Philadelphia’s excise tax on sweetened beverages, where cross-border purchasing led us to confront interference and develop methods to better inform policy decisions.

Through these examples, I will reflect on how statisticians can seek opportunities to bring their expertise to consequential real-world problems, and on the importance of collaboration, communication, and stepping outside our usual academic boundaries. Most importantly, you don’t have to choose between being a methodological statistician and having an impact outside statistics.

This looks interesting. Click on the link above to register for the talk and get the zoom link.

“I write to seek your help and advice regarding a published article that my group and I believe is invalid and should be corrected or retracted.”

Posted on by
9

David Allison writes:

I write to seek your help and advice regarding a published article that my group and I believe is invalid and should be corrected or retracted. Despite our concerns, the editor seems to be unwilling to accept responsibility and to be responding to us in a way that we perceive as unethical.

You may wish to see a full account on this matter including the Letter to the Editor (LTE) that was rejected by the journal [Di Molfetta et al LTE (short version) Submit.pdf (attached)], our email exchange with the editor following its rejection [Diabetes Research and Clinical Practice Editor Email Exchange.pdf (attached)], and our email exchange with the authors and editor prior to submitting the LTE [Di Molfetta Email Exchange.pdf attached)], but a succinct summary of the events are provided below for brevity:

07/30/2023: We identified a paper as having errors related to not properly analyzing cluster randomized trial data.
08/08/2023: We contacted the original study authors and requested the deidentified data to reproduce the published results.
08/21/2023: The authors requested clarification for the reason for the data request.
08/21/2023: We provided clarification for the data request.
09/22/2023: We prompted the authors to reply to our prior communication.
09/30/2023: We included the editor in the email exchange requesting that they uphold the journal’s data sharing policy.
10/14/2023: We prompted the editor to reply to our prior communication.
10/14/2023: The editor replied and asked for clarification for the reason for the data request.
11/03/2023: We clarified that we requested the data to reproduce the analyses and re-analyze the data using the correct statistical methods for the study design. We also provided a summary of the errors we identified.
11/23/23: The authors replied with their rebuttal to sharing the data, rationalization for their errors, and rationale for not correcting them.
04/29/24: We submitted a letter to the editor to the original journal.
05/01/24: The manuscript was rejected on the basis that our “paper cannot be assigned sufficient priority to allow publication.”
05/08/24: We wrote to the editor directly to reconsider publication of our letter.
05/09/24: The editor responded that the journal does not wish to be engaged in a legal issue and that the matter will not be entertained by the journal any longer.
05/10/24: We responded to the editor to clarify that we simply want the journal to publish our letter and had not proposed any legal action.

The editor has not responded to our last communication as of 05/21/24.

We believe that the most recent response from the editor is patently unethical and incommensurate with the publishing standards of Elsevier, guidelines of COPE, and generally accepted norms in the field of science.

We have two requests:

1. Do you have any advice for us on how we might best pursue this issue further.

2. Would you be willing to blog about this item? If so, you might compare and contrast it with another recent case we pursued [BMC Public Health LTE Example.pdf (attached)] which was an exemplar situation where the journal was responsive to the ethical issues we raised.

I don’t have any advice on what they should do! The flood of bad research is just so overwhelming. But I can blog it. So I did.

David Allison often works in nutrition science, an area where our government is spreading misinformation that is then sane-washed in the news media. And, along with all that, there’s so much uncertainty and variation that it’s hard to know what to think or do.

Why do we like wind chimes?

Posted on by
7

Dmitri Tymoczko writes:

You know your question, “why do we like to listen to music over and over again?”

I think the question reveals that our relation to music is sensory and pleasure-based rather than cognitive and information-based. Music (like food and alcohol and sex) gives us pleasure, which is why we do it over and over.

Anyway, I am writing because there is a mirror version of the question: “why do we like wind chimes?” Wind chimes are essentially random notes, but people like them. We don’t really like random words, because our relation to words is information based.

I wanted to title this post, “Why do we like to ask, ‘Why do we like to listen to music over and over again,’ over and over again?”, but I decided that was too cute so I switched to the more SEO-friendly version above. Everybody likes wind chimes, right?

To answer Dmitri’s question, my personal theory of “Why do we like to listen to music over and over again?” is that there’s a big difference between listening to music actively (or playing music), and listening passively (which is how I usually do it).
Continue reading →

Seeking to interview social scientists experimenting with LLM simulations of opinions or behavior

Posted on by
13

This is Jessica. As part of our interest in how social scientists are using LLM simulations of human behavior, some collaborators and I are seeking interview participants. We are thinking about tools to support rigorous design and evaluation of such studies, and would like to hear examples of goals and challenges that researchers who are experimenting with LLM simulations are facing.

Specifically, we’re looking for researchers in academia or industry who are currently using LLM simulations in their work and could bring a simulation dataset (and optionally, human ground truth data) that they are working with to the interview. We are interested in use cases like piloting surveys or experiments with LLMs, replacing or supplementing human participants in surveys or experiments with LLMs, or using LLM simulations for post-study analysis or theorizing. The interviews should take no more than 1 hr.

Email me and Xudong Tang ([email protected]) if interested, and/or pass this on if you know of social scientists who might be.