Kharon Applied Intelligence Fellowship in Denver, London, or Madrid

Juan Lopez Martin (coauthor of the very useful MRP Case Studies) writesL

Kharon, where I work, is launching a new program called the Applied Intelligence Fellowship, and I think Master/PhD students at PoliSci/SIPA/Stats/QMSS would be a great fit.

Kharon operates at the intersection of data science and global issues, building data, intelligence, and technology products that help organizations understand geopolitical and economic security risks. Our work maps the connections between companies, people, technologies, supply chains, and networks to reveal relationships with real-world consequences.

The fellowship is a six-month, full-time, paid program for people interested in using AI and computational methods to understand complex global systems, geopolitics, and economic security. I’ll be working closely with the fellows throughout the program, which is also designed to lead to a permanent role on our team.

I just noticed that the deadline for the application is 23 Nov, so if you’re interested, you better get to it right away!

Major publisher releases AI-written history book that is full of glaring errors

There’s this rhetorical trick where you defamiliarize a situation by putting it in another context.

For example, the other day, in a post titled “Hunter Biden’s assets under management at his investment fund went from $200 million to $3.5 billion in the first year of his father’s presidency,” Paul Campos wrote:

Excuse me, but there may be some sort of inaccuracy in this headline . . . Financial Times reports that Donald Trump Jr.’s venture capital firm, 1789 Capital, has grown from roughly $200 million to $3.5 billion in assets over the last year, while pitching an “America First” investment strategy centered on “patriotic capitalism.” The Palm Beach-based firm says it aims to reach $10 billion in assets and has invested heavily in AI, defense, and tech companies including Databricks, Groq, xAI, SpaceX, and Anduril. One partner told the FT the firm hopes to become a new Carlyle Group-style powerhouse, using deep political connections in Trump’s Washington to build influence and attract capital.

Campos’s headline does useful work here, giving us a sharp perspective on what would otherwise be a sadly routine story of political and business corruption. The feint in the headline implicitly points us to the important idea that this sort of influence is not just the product of damaged and selfish individuals (“Hunter Biden”).

I want to do the same thing here, hence the title of the present post. A more accurate title would be “Hallucinated history from humans.”

Here’s the story. Reviewing a new book, “A Short History of America,” historian Nicholas Guyatt writes:

The book has a distracting number of errors. Some are cosmetic or minor: perhaps only nerds like me will grouse that the Confederation Congress, which governed the US before the Constitution, appears here as the “Confederacy Congress”. A few more readers will protest that the division of Korea took place in 1945, rather than 1948, and that the percentage of African Americans who were free rather than enslaved before the Civil War peaked at 14 per cent, not 40 per cent – a crucial distinction in terms of the long struggle for equal citizenship. . . .

But it really matters to American history that no colony abolished slavery before the Revolution. . . . And it would be hard to agree that the disastrous US intervention in Somalia, which inspired the movie Black Hawk Down, “wiped out [George H. W.] Bush’s electoral popularity” when Bush sent troops to East Africa in December 1992 – more than a month after he had lost the presidential election to Bill Clinton. . . .

I looked up Simon Jenkins, the author of the book in question, and he’s a journalist in his 80s, so, sure, it stands to reason that he’ll make mistakes if he whips off a book on a subject on which he’s no expert (according to Guyatt, the book was itself based on previously-published overviews with no reference to any primary sources). The interesting question is not how an elderly unqualified author can write a book full of errors, but why it would be published (I guess it’s readable so the publisher thought it would sell books? And I guess the author didn’t ask for a lot of money for it?) or why it would get a serious review (I guess because it was put out by a serious publisher?). I wonder who will buy the book. General readers? Or is it intended for schools? Lots of school history textbooks are notoriously bad.

What about the connection to the title of this post? My point is that the author of the error-filled volume is hardly relevant here. The active participants are the publisher and whoever is willing to shell out £25 for the book. Whether they use a chatbot or a hack journalist (albeit with a classy English accent and a “Sir” attached to his name) hardly matters, especially given that the journalist in question seems to have written the book by a chatbot-like process in which he reads a bunch of existing books and then regurgitates them while offering his own spin.

As we’ve discussed elsewhere, our experience with chatbots gives us a fresh perspective on how we often think and behave as humans.

Can you help to identify psychology’s most important unanswered questions? Each winning question receives $400.

Igor Scaldini writes:

I am a research assistant at Clearer Research, where we are running the Unanswered Questions in Psychology Prize to collect the open questions the field most needs to answer. Each winning question receives $400, and winners are chosen by a vote of current and former academic psychologists.

An example of the kind of question we mean:

“What are the true psychological impacts of social media, and how do these differ by demographic group and by social media app type?”

Submitting takes about ten minutes, and the deadline is December 31, 2026.

I know these people—I was recently interviewed by Spencer Greenberg for their podcast—so I think the $400 thing is real.

On Spy Fiction

Following a review in the newspaper, I picked up The Collaborators, a spy novel by Michael Idov. I read and enjoyed it, but it was just OK, not as great as I would’ve expected from the review. So, that’s fine, tastes differ, but in this case I thought it would be possible to go deeper, by considering more carefully what I expect to see in a spy novel.

Like all genre fiction, spy stories have their conventions. But maybe because the spy form is a bit narrower than, say, crime fiction or science fiction, its conventions are more constrained.

What follows is a list of the expected features of spy fiction. Most are positive from my perspective as a reader, some are negative, others neutral. But I consider all these to be typical features of the genre.

Before going on, let me situate my tastes. My favorite spy novel ever is Tinker Tailer Soldier Spy, I’m also a big fan of Eric Ambler and Graham Greene (not a genre writer like Ambler or Le Carre but he did write some classic spy stories), and my favorite modern spy novels are the first two Tourist books by Olen Steinhauer. I also love the Slow Horses books—yeah, call me a middlebrow, I don’t care! I’ll refer to some of the above literature when discussing the features below. And I also enjoyed Black Bag and The Americans, but I won’t talk too much about movies or TV in this post.

Disorientation

Spy novels are like science fiction and crime fiction in that they get much of their power by inducing in readers a feeling of disorientation. In science fiction, this might be a future or alien society or just something unexplained entering our normal world, a puzzle to be solved. As for crime and detective stories, they’re called “mystery” for a reason. You start with a pleasurable feeling that you don’t know which way is up, and then, as you read, various mysteries get resolved and new mysteries surface—maybe enough to give room for a sequel. (I heard an interview with Mick Herron where he said that he purposely introduces mysterious elements to his books without them being plotted in, just to give him room to maneuver in future books. Kind of like those one-story houses with rebar sticking out, in preparation for the eventual second level to be built.)

You might think that there’d be no point to re-reading genre fiction: once you know all the spoilers, the suspense is gone. But the fun can go the other way too: reading the story without the initial disorientation is an entirely different experience. It’s actually pretty amazing to read Tinker Tailor, after knowing how the story is going to play out.

Disorientation is harder to pull off in a series. The Slow Horses books take place in a pretty constrained universe, and it doesn’t take long to realize that Jackson Lamb is a good guy, etc., so in this series the disorientation has to be created from the new plot for each individual book. Which is fine but it’s not the full experience. The Tourist series achieves continuing disorientation by upping the stakes each time, but after the second book, the unfortunate result is that he’s kind of run out of rope. In retrospect I think it was a mistake for Steinhauer to have killed off the Tourist program at the end of book 1. It was fine in that book, but it gave him less to work with going forward. In contrast, Herron does much better at keeping the stakes high in each book without escalating.

Mildly exotic surroundings

Much of The Collaborators takes place in Latvia and Moscow, and, yes, these are exotic locations—but not too exotic. As readers, we have some idea of what to expect in post-Soviet landscapes.

This is par for the course: just about all the spy novels I’ve read will put the characters in an interesting locale—indeed, sense of place is an important aspect of spy fiction, which makes sense given the connection between spying and nationality—but it’s usually not anything too unfamiliar. I’m not sure why it should be the case, except that spy stories are by their nature international, but it seems like a pretty consistent pattern to me.

Moral seriousness

At the simplest level (for example, Graham Greene’s early novel, The Confidential Agent), a spy story is about confounding the knavish plots of the bad guys. Then there’s mid- and late-period Eric Ambler, where both sides might be amoral schemers, but the reader is still made aware of the stakes: innocent lives are involved. Le Carre and his successors take it to the next level, where even the good guys are reflecting on their culpability. But, even without that sort moral ambiguity, it can’t just be a game. There has to be a purpose to it. One reason that Chris Pavone’s The Expats left a bad aftertaste to me was that I didn’t feel that air of seriousness. Yes, there was peril, and I cared about the characters while the story was playing out, but it seemed more like an adventure than something important, and when it was all over they just seemed like a bunch of killers to me, and I was annoyed that I was supposed to be rooting for them. In contrast, Slow Horses is a lot of fun, but there’s always an underlying sense that they’re doing this for a reason.

Sexism

Not a good thing, but this is just the tradition in most of literature. Even more so in espionage, where the overwhelming majority of real-world agents have been men. But it’s not just that. Le Carre’s casual sexism has been long noted, and it’s taken a while for authors in this genre to catch up to modern sensibilities. Even the recent book by Michael Idov has trad tropes such as referring to the male lead by his last name and the female lead by her first name, also various descriptions of female characters as being amazingly beautiful. Can’t a woman just be attractive to some guys—does she always have to be such a looker (or the opposite, as with Tinker Tailor’s Connie Sachs)? The recent spy novels that best handled this, in my opinion, are the Tourist books. I have a feeling that Steinhauer had to do some “affirmative action” here, populating his story with more major female characters than would exist in real life, but it worked for me. Similarly with Slow Horses: I get the impression that Herron worked hard to get enough woman characters into the mix, and I also think that the effort paid off.

Good action scenes

People don’t always think about this for books (as opposed to movies), but there is such a thing as a compellingly-written action scene. As in a movie, you need emotion, micro-suspense, the parallel placement of action and thought (i.e., you get a sense of the characters’ thinking and decision making while the action is happening), along with some legibility of who’s doing what when, who’s in the scene, what might come next, etc. Also a delicate balance between excitement and plausibility: on one hand, you don’t want to just see some superhero punching everyone out; on the other, a bit of fancy footwork is appreciated.

I’m no expert on how to write a good action scene, but I can recognize it when I see it. The Tourist books have some great action scenes: the bit where he breaks into the Paris apartment in book 1, the art heist in book 2 . . . just so well done. The Slow Horses books have lots of good action and chase scenes. They’re not uniformly perfect—some verge on slapstick—but the best scenes in those books are a hundred times better than Jack Reacher mauling some bad guy. Le Carre also does it well. He doesn’t have much straight-up violence, and when he does, he doesn’t do it so well, actually, but he has compelling action scenes. For example, the episode in Tinker Tailor where Peter Guillam is sneaking the file out of the Circus library, that’s done perfectly.

Eric Ambler’s action scenes aren’t bad, but most of the drama in his books comes from the explanations, not the action. And I don’t recall Graham Greene being particularly good at action scenes either. Here’s a theory: Good action scenes came originally from the movies, and then authors figured out how to capture the same feeling in prose. Not that movies always do it well; for example, the action scenes in John Wick seemed pretty bad to me. And, for that matter, Star Wars had excellent special effects but poor action scenes, just a bunch of faceless baddies aiming their blasters at Luke and missing.

I was not impressed by the action scenes in The Collaborators. There’s a car chase in Moscow that’s not bad but nor is it so great, and the shootout in the building where the hero works isn’t done well at all. It was just the good guys being heroes.

The interesting thing is that it’s hard to write a good action scene, in the same way that it’s hard to direct a good action scene in a movie, where you have a sense at all times of who’s doing what, where the weapons are, what options the characters have, etc. An action novel that works is similar to a movie with good production values, which doesn’t necessarily mean a big budget, but it does require some care and some respect for the audience—no Netflix-style narration of the action, but rather a sort of compact with the reader.

Moral ambiguity

One thing that distinguishes Cold War and post-Cold War spy novels from other forms of genre fiction is that there’s almost always a sense, not of moral equivalence exactly—you’ll typically get the sense that we’re the good guys—but a sense that spying and deceit take you to the edge of morality, and that it’s important that “we,” in fighting “them,” don’t “become” them. In early Graham Greene and early Eric Ambler, it was simpler—we’re defending ourselves from the fascists—but after the end of the second world war, there was no clear line. Even when we’re fighting against universally-recognized baddies such as ISIS or Putin’s Russia or whatever, there’s some sense that (a) there are some reasonable people on the other side, and (b) sometimes our side uses inappropriate tactics of “the cure is worse than the disease” variety. Just about all the spy novels I’ve read have this sense of ambiguity, not quite a symmetry between the two sides, but a sense that each side has some good and some bad that can’t really be untangled. I’m sure there are some spy stories that are straight-up cops and robbers, but those wouldn’t be the sort that I would read.

Tradecraft

One of the great pleasures of spy stories is how we get convincing and seemingly realistic descriptions of surveillance, dead drops, jargon, and so forth. All the spy fiction we’re discussing here does this. Le Carre is the master here, but all the others do it well. It’s the price of admission. No tradecraft and it’s not really part of the genre.

Betrayals

This is another one that goes with the territory. Traitors and moles, also intelligence services that sacrifice their own agents (as in The Spy Who Came in from the Cold), all sorts of backstabbers. Some of this comes straight from reality, with Tinker Tailor being a sort of retelling of the Cambridge Five, but, even more than that, it’s inherent in the subject matter. To spy is to act deceptively.

Plot twists

A twisty plot is not a necessity in a spy novel. The Spy Who Came in from the Cold and Tinker Tailor had twisty plots, but some of Eric Ambler’s novels are pretty straightforward. Their plots have surprises and revelations but not big Gone Girl-style twists that cause you to reevaluate everything that came before. The thing with plot twists, though, is you have to manage them well. If you, the reader, can anticipate the twist before it happens, that’s death to the plot. The Collaborators failed for me on those grounds. I couldn’t see everything that was coming, but the big twists were telegraphed.

Loyalty to people versus loyalty to country

This one’s essential. Whether the hero is an insider or a freelancer, a bureaucrat or a cowboy, a Clint Eastwood type or a sensitive new age guy, he has to choose between his friends and his country, and it’s essential that this not be an easy choice.

The hero is an underdog

Underdogs are sympathetic characters. James Bond is not an underdog, but he’s the exception that proves the rule. John Le Carre, Len Deighton, etc., have all been characterized as writing anti-James-Bond novels, but at this point Ian Fleming is the outlier, and when I think of spy stories, I don’t think of James Bond—yes, he’s a spy, but I think of his adventures as an entirely different genre, or, should I say, a form of male wish-fulfillment that’s less interesting to me.

I’d say that most literature puts in some effort to establish the hero as an underdog, starting off behind the eight ball and needing to overcome various challenges. The Collaborators does this too, but not in an effective way. I was somehow reminded of the movie Swingers, which I loved, but which did the annoying thing of having its hero walk around wearing a Queens College sweatshirt. What a man of the people, huh? In real life, though, the actor in question, who was also the author of the screenplay, did go to Queens College! His working-class cred was legit, but I was so used to poseurs that when I saw the sweatshirt in the movie, I cringed.

Novels, not short stories

Successful spy stories are almost always novel-length. Maybe because, in addition to setting up the plot and characters, a good spy story creates its own world, a parallel universe with the Secret Service, international conspiracies, and so forth. Tinker Tailor created the mythical Circus (yes, I know this was not the first book set in the Smiley universe, but the previous volumes in the series gave you very little backstory), Slow Horses set up a world-within-a-world of British intelligence, The Tourist created this cool clandestine program, and so forth. Indeed, one reason The Tourist series started falling apart after the second volume was that it set up too much of its backstory at the beginning, and then it had nowhere to go but to escalate.

But is the espionage genre special? Novels dominate short stories in all areas of fiction and have done so for a long time, at least since the decline of the sorts of mass-circulation magazines that published the stories of F. Scott Fitzgerald and others in the mid-twentieth century. After the war, the Golden Age of magazine science fiction stories was nearly a decade old, and the Golden Age of mysteries was mostly in novel form; indeed, you’d have to go back to the era of Sherlock Holmes to find a time that short stories were dominant.

With literature, all is different now: fiction is no longer a form of mass entertainment, so the goal is to appeal to niche audiences, not to grab the commuter who will buy a book off the rack at the train station. In any case, for the foreseeable future we can expect the written spy story tradition to continue in the form of novels.

Cynicism

This is kind of related to the themes of moral ambiguity, underdogs, and betrayal mentioned above, but slightly different that it’s more a matter of tone. The hard-bitten hero has his pangs of conscience and he cares about the cosseted civilians he’s protecting, but nothing surprises him. This attitude seems pretty much universal in spy stories. The only exception I can think of is the movie Three Days of the Condor, where Robert Redford seems legitimately shocked by the bad things going around him.

Balance between realism and fantasy

This one’s tricky. On one hand, a key part of the appeal of spy fiction (again, James Bond aside) is its verisimilitude, the sense that you’re getting a look into a gritty aspect of the world that you wouldn’t otherwise know about, indeed something closer to the “real world” than what we see every day. It’s the opposite of Agatha Christie—nothing against her books, they’re great, it’s just that she deliberately puts you in an artificial world (as do Conan Doyle, Rex Stout, and all the rest), whereas the setting of spy stories, even when clearly made up, are supposed to represent something about the structures underlying our world.

On the other hand, as readers we want excitement and good production values; see “Good action scenes,” above. The Americans was convincing in many ways, but it wasn’t so plausible that these sleeper agents would be karate-kicking their way through death-defying fights every few weeks. Le Carre did the best with keeping the action plausible, but a bit of fantasy in a novel is fine too. I enjoyed the implausible elaborate schemes in Eric Ambler novels and the stunning action scenes in the Tourist novels. But The Collaborators went too far in the fantasy direction. At one point the hero beats somebody up in a public place for just about no reason, and everybody’s like, OK, that’s cool, he’s just a passionate guy. This was more like a Sylvester Stallone movie than what I’d expect to see in a spy novel. Actions should have consequences.

Serviceable writing

In my reading, Graham Greene is a better writer than Eric Ambler, who in turn is a better writer than John Le Carre. By “better,” I mean deeper but also just more fun and interesting to read the sentence and paragraphs. But Le Carre (and the other writers named above) are just fine. Even Michael Idov, who has some really clunky prose—I blame this in part on the disappearance of editing in the modern publishing era—it doesn’t really matter. Some of the notes are jarring but ultimately that’s minor in my judgment of the book. The purpose of spy-novel prose is to convey the action and the story, and, for this, the writing just needs to be above the bar. In that way, it’s similar to other genre fiction: when you get excellent prose, that’s a benefit, and it can make up for flaws elsewhere in the book, but it’s not a necessary or characteristic feature of the genre.

Protecting one’s own comes first

We were joking about this after watching a bunch of episodes of the excellent TV series, Le Bureau des Légendes: The main job of the intelligence officers in the show seems to be to clean up after their own mistakes, rescue their agents in peril, etc. But, come to think about it, that’s so often the case in spy stories. Tinker Tailor is all about finding the mole whose biggest crime is to have betrayed his fellow operatives. The Tourist is mostly about the main character trying to stay alive, while the baddies are out there killing other agents. The heaviest emotions in the Slow Horses series are when Lamb is protecting or avenging his “joes.” And so forth. The Collaborators follows this pattern: what upsets the hero the most is when his spy buddies get whacked, and most of the action is about him avenging them and protecting his girlfriend. It’s all a kind of circular thing where the justification for the intelligence agencies is that they are needed to protect their own officers.

Anything else?

There are probably some other characteristic features of spy novels that I’m missing.

Postdoc opportunity! Here with me at Columbia! Cutting-edge Bayesian analysis and causal inference: Statistical Tools for Generalizing in Education Research.

We’re seeking excellent applicants who are excited to make important contributions in Bayesian modeling, statistical computation, education research, and quantitative social science, in a fun and demanding work environment.

The project aims to develop new statistical tools for nonrepresentative samples and imbalanced experiments, addressing problems of generalization that are central challenges in empirical research. The project will improve statistics and research methodology in education research by enabling researchers and policymakers to make more reliable generalizations and focused estimates, unifying the ideas of multilevel regression, poststratification, and sampling weights. The goal is for education researchers to be able to fit more sophisticated regressions, multilevel models, and small-area estimation to real-world data.

The approach will use the following statistical methods: Bayesian inference, sampling theory, bootstrapping, Monte Carlo methods, multilevel regression, and poststratification. The core of the proposed method is a new theoretical idea for embedding sampling weights into a joint distribution with modeled outcomes, along with analytical work mapping population to sampling distribution, and a computational implementation using probabilistic programming. The postdoc will help apply these methods to existing experiments in education research, including the Head Start Impact Study.

The postdoc will work with Prof. Andrew Gelman in the Department of Statistics and Political Science at Columbia University and be immersed in an active research community, including Ph.D. students and postdoctoral researchers in statistics, economics, political science, and biostatistics; programmers; and a weekly meeting on causal inference. The project will involve active collaboration with Prof. Tyler Watts of Teachers College, Columbia University. In addition to the immediate research group, the Departments of Statistics and Political Science and Teachers College are lively research communities involving a wide range of projects, and there will also be the opportunity to work with others at the university and in the larger New York City research community.

Review of applications begins immediately and will continue until the position is filled.

You should have a PhD at the time of appointment, expertise in Bayesian modeling and computation, and an interest in education research. Knowledge of survey sampling and causal inference is a plus, and the work will involve programming in Stan, R, and Python.

Please send a cover letter (explaining your interest and qualifications for the position), a C.V., and names of at least two recommenders to me at [email protected]

Survey Statistics: Blue Rose Research is hiring !

As readers may know, I’m a survey statistician at Blue Rose Research. We survey the public to forecast elections and test political messages, used to advise Democrats. We are hiring for several roles heading into the 2028 election cycle: https://blueroseresearch.org/jobs/

All positions are remote, with an option to be in person in New York City.

Please circulate and apply !

The polar bear is at Blue Mountain Reservation:

 

The 2026 Pangram social justice misinformation wars: Collective understanding of binary classification takes a hit

This is Jessica. I continue to be amazed by how much confusion there is about Pangram, the AI text detector. As Sam Kriss reports on Substack, not only are people sharing misinformation amongst themselves, the idea that “Pangram is racist” and results in high false positive rates for African or Caribbean or non-native English speakers is worming its way into the record, unbacked by any solid evidence. AI search results are now promoting assertions about bias that are challenged by existing audits of Pangram on texts from non-native English speakers. Academics are calling such tools racism clothed “in a veneer of objectivity.” Motivating such accusations are several cases where POC short story award winners won with texts that Pangram flagged as 100% AI-authored. The result is a dent in our collective understanding of basic binary classification.

With all of this apparent controversy, it’s reasonable to wonder what is going on. Let’s review what is actually known about Pangram’s error rates, and then consider why so many people are buying into the attempts to make this a social justice issue. 

But first, if you read no further, the TLDR is that false positives (where Pangram flags human-generated text as AI-generated) on texts of reasonable length (at least 50 words, with greater reliability as length increases) are extremely unlikely according to every general audit I could find – the largest report FPRs on the order of 1 in 10,000 for audits on mixed length texts. In fact, Max Spero, Pangram CEO, posted on X a few days ago that based on a bounty program, they had found their first false positive on a long text created before 2020: a poem that repeats the same six words four thousand times. False positive rates on non-native English speaker corpora remain indistinguishable from these overall rates according to large audits. This doesn’t mean Pangran is reliable on every distribution of human-authored text, adversarially generated or not. But the evidence of low false positives on natural distributions of human texts is much stronger than the cherry-picked data points the racism-accusers are wielding. 

Audit results

According to the latest model card published by Pangram itself, on Pangram 4.1 released October 19, 2026, overall false positive rates are:

Note that Pangram uses a strict definition of false positive: if a known-to-be human written document has any segment classified as AI-generated or AI-assisted, the entire document counts as a false positive. Similarly, a fully AI-generated document with any segment labeled human or AI-assisted is counted as a false negative.

Overall false negative rates are higher than false positives (by about 40x) but still quite low: 

The above makes clear that error rates are asymmetric. This is intentional, and something a model developer can control, by using a training objective that prioritizes avoiding false positives or more directly by setting a conservative classification threshold learned on held-out data. This makes it possible to ensure a very low FP rate on text generated similarly to the audit distribution (Pangram describes targeting 1 in 10,000, and available audit evidence is consistent with this). 

When Emily Bender says that “As is widely known within the NLP research community, perfect results usually point to something wrong with the evaluation setup”,  she’s either deliberately ignoring or unaware of basic statistical machine learning methods.

Other, smaller audits of Pangram models include Epoch AI’s from this past summer, which found that Pangram produced zero false positives on 495 human passages. It detected all of 297 AI passages created using basic prompts, but missed 30 of 297 (10.1%) when the LLM was prompted to imitate a particular author’s style. Another by Karr, Khvatskii, Hua, and Chawla from last month finds that Pangram had zero false positives on 306 pre-LLM abstracts (2013-2015), but it flagged 14.9% of recent abstracts (from 2023 – 2025) as AI-generated. However, the authors don’t know what the ground truth was on the recent ones, so this statistic cannot be treated as a false positive rate. A very small audit by Van Vlasselaer, Van Droogenbroeck & Spruyt from June, which found that Pangram detected 97.5% of 40 fully AI-generated papers and about 95% of hybrid human and AI generated papers and humanized AI-generated papers. In one of the first external audits in 2025, Imas and Jabarian found that version 2 Pangram achieved nearly zero false positives and false negatives on medium and long texts.

Where is the evidence that Pangram is racist?  

I am not aware of any audits with known ground truth specifically focused on texts by POC, but there are a few related analyses. Vauhini Vara reports that when she ran 25 prior prize winning stories of the Caine Prize for African Writing through Pangram, all came back 100% human-authored.

Adding to this, yesterday Tuhin Chakrabarty posted an analysis of books written by African authors: 97 from Zimbabwean authors written 2020 to 2025 and 90 from other African countries written from 2020 to 2026. He found just one that was flagged at least 1% AI by Pangram.

Beyond the claims that Pangram is racist against African or Caribbean authors, there is a general line of attack that it should not be used because errors rates are unacceptably high for non-native English speakers. Yet Pangram has produced multiple rounds of audits using four non-native english text corpora.

FP rates for version 3.3:

Hardly higher than the overall rates reported above. 

An audit of version 4 compared to 3.3 shows lower rates:

And an audit of 4.1 from October this year on 15,420 documents by English-as-a-second-language undergraduate and graduate students produces literally zero false positives:

So why are so many people confused?

Here I can only speculate. 

Incentives for organizations to save face. Both the Commonwealth Prize and Caine prize organizations report doing investigations that supported the accused authors’ case. If you’re looking to the literary authorities to tell you what to think, it seems that false positives are occurring repeatedly for long texts by POC authors. However, neither has made any of the evidence they claim to have gathered public, so believing that the stories are false positives means trusting their word. 

Misinfo to protect AI habits. Blatant misinformation is certainly being propagated, often by people who appear to want to keep their current productivity edge by having AI write everything for them. Here’s just one example of someone posting a faked Pangram false positive screenshot on X: 

Confusing FPs and FNs, confusing different classification tasks. Some may simply be mistaking overall accuracy or false negative rates as evidence that Pangram is not reliable when it flags something as AI-generated. There also seems to be some confusion about what Pangram does – detect who strung the words together – versus other forms of detection we might want, like being able to tell who came up with the ideas. But this is not what Pangram targets – if you use AI to rewrite human texts, like rewriting abstracts in the Karr et al study above, and you don’t take steps to constrain the AI from only fixing typos or other errors, then yes, Pangram will flag it as highly AI-generated, because that is what Pangram it is designed to do. 

There are also people arguing against use of Pangram based on error rates when texts are generated adversarially, like examples that were specifically written to sound like AI, in contrast to the audit results above which apply to naturalistic distributions of human-generated text. When we target a particular FPR for a classifier, we expect that to hold only so long as the training or calibration data we used is representative of the cases we encounter in practice. 

Finally, how to interpret Pangram percentages can be confusing, and attributions of the source of small pieces of text within a larger text are not covered by the audits above. Unreliability in identifying which portion of a text contributes a high score can suggest which use cases to be careful with, but it doesn’t invalidate the document-level false-positive estimates from the audits. 

Splashy earlier results on bias in other detectors. A early (2023) Stanford study by James Zou and others did claim that GPT detectors are biased against nonnative English writers, but this was on a very small sample (91 essays from theTOEFL exam) and did not test Pangram.

Inaccurate folk theories. Another contributor might be inaccurate folk theories about what Pangram is. Maybe some people imagine Pangram doing something like they would do if they were given a text and asked if it was AI-generated – e.g., look for cues that are said to be associated with AI text like m-dashes and decide based on that. There are some stories consistent with this, like where Granta turned to a Claude instance to help them decide if AI had been used rather than a trained detector. In reality, Pangram is a classifier, trained with lots of labeled texts to discriminate between which are human-generated and which are AI-generated, and calibrated to keep the false positives low on average in naturally generated texts.

Concern that detectors are more often applied to POC and ESL writers. Maybe people are perceiving real asymmetries in whose work is being checked, and reacting to that by trying to discredit the tool.

Resisting AI as a solution to anything. Finally, I think a lot of people resent the imposition of AI in their lives, and see relying on an AI detector like Pangram as accepting the inevitability of AI, which they want to resist. So they say things that imply Pangram is unreliable for all purposes without realizing where the evidence contradicts their assumptions.

On the larger question of whether we should care about AI writing 

We need to be much more careful in distinguishing between the evidence on false positives and the broader question of whether it’s a good idea to try to detect AI writing. Everyone is entitled to an opinion on the latter, but the former is better left to the statisticians. If you want to avoid using Pangram on the grounds that you believe the correlation between who wrote the words and who controlled the ideas is so low  that your decision quality will be no better or worse, then I can respect that – we have different priors. Or if you personally want to avoid developing a habit of relying on AI detectors because you believe that human writing will become more and more resemblant of AI writing with time, or AI-written text less indistinguishable, that’s fine, it’s up to you how you make decisions for yourself.

What I really dislike is the way that people (including academics like Bender) are adding confusion by ignoring (and even actively arguing against) how a positive classification from a classifier like Pangram can have evidential value for some of the decisions people care about right now. Sure, false positives are possible, but this does not warrant recommending to people that they remain in ignorance on every case, even if they might benefit from some signal on who wrote the text. Sure, distributions can shift, but there is a difference between recognizing that estimates may not be calibrated in a new context, and concluding that existing evaluations are “effectively meaningless.” It’s like all these Pangram-haters want to scare everyone else into being as irrational as they are by preying on their fear of being labeled racist. 

More on the rationality of using a signal when it is informative here.

Algorithmic deepfake science

1. Chatbots

There’s been a lot of talk lately on the routine use of chatbots as a scientific tool (literature review, refactoring code, checking the internal logic of an argument, etc.) as well as the possibility of directing chatbots to perform investigations and write entire papers.

Two big ideas come up:

(a) Chatbots being used to create “deepfakes” of scientific research: papers that are professionally formatted, readable, and are written in the style of top research papers—simulacra of research that is convincing enough that it can get published in serious journals. Such papers could include plausible computer-generated images and data, along with chatbot-generated text describing nonexistent experiments.

(b) Chatbots being used to make real discoveries in pure math or computer science, or doing automated analysis of real data in biology to make real scientific discoveries.

Both these ideas are versions of things that chatbots, generative AI, and machine learning can do well: (a) producing text that looks like human-generated text and producing images and videos that look like real photographs and movies, and (b) finding patterns in big data.

2. Deepfake science conducted by humans

But today I want to talk about something slightly different, which is the pre-AI history of deepfake science, that is, deepfake science conducted by humans.

Let’s go through some history. I don’t claim this to be a systematic history; it’s just some examples I’ve heard about over the years:

– Old-style hoaxes: physical deepfakes such as Piltdown Man, fake data such as from the psychologist Cyril Burt, the doctored fairy photos that fooled Arthur Conan Doyle, the Bell Labs superconductor guy, etc.

– Fraud from the recent golden age of NPR/Ted-style fake science: the Harvard primatologist, the Cornell food-behavior guy, the election-denying gun guy, the experiment-faking UCLA poli sci student, the Surgisphere team, the UNR dean of engineering, etc.

– Data misrepresentation or plagiarism: the University of California sleep guy, the USC giraffe guy, the Stanford president, whoever at Columbia faked the U.S. News report, a few zillion p-hackers, and various researchers who, when confronted with problems with their data, acted as if there was nothing wrong.

– Bad science, not involving fraud but being “deepfake” in the sense of following the detailed forms of a scientific investigation without actually making scientific progress, because the data and measurements are too noisy to learn anything about the target phenomenon (what we call the kangaroo problem): the beauty-and-sex-ratio guy, the ESP guy, the air-pollution-in-China guy, the ovulation-and-clothing team, the “critical positivity ratio” team, the climate-change “gremlins” guy, the mind-body-healing team, etc.

I’d call all of these “deepfakes” in that they are simulating the process of science in a generative manner, not constructing a scientific paper by itself but performing a (human-operated) process that yields a convincing-looking result. As we’ve learned with computer-generated text and images, it can be much more effective to run such a process than to just try to jump to the end.

Depending on the researcher—or, maybe I should say, the operator–the generative process can be done in different places: text, data, numerical images, even the theory itself.

You can go pretty far with a deepfake but eventually you’re likely to be caught out, either because your fake data have internal flaws, because your results can’t be replicated (as with the superconductor), or because there’s a problem with the “meta-data” or cover story. Consider for example the surveys purportedly conducted by Mary Rosh and Michael Lacour: in these two (separate) incidents, no documentation was produced. Or Wansink’s refilling soup bowl and Ariely’s paper shredder: the claimed physical evidence for these devices is dubious.

As the saying goes, reality is overdetermined, and ultimately the only convincing way to make fake data is to collect real data, in the same sense that the only realistic map of a territory is to use a Borgesian 1-to-1-scale.

3. Algorithmic deepfake science

Stepping back, we can think of both the human-generated and chatbot-generated forms of deepfake science as being algorithmic, in the sense that they are created using some sort of general process that is being fed by domain-specific information (not so much information about the science; more information about the conventional forms of data and writing used in the particular scientific subfield being faked).

To put it another way, this work is “deepfake,” rather than just “fake,” in that it’s generating a simulation of the entire scientific process, creating a thick brick of fakeness, as it were, rather than a brittle fake crust. Bem’s ESP edifice, for example, isn’t just a convincing-looking paper; it also includes, at an internal level, experiments, data and a research team, and, externally, a scientific literature and repeated exposure in the news and social media.

Last year I posted 7 steps to junk science that can achieve worldly success:

1. Bad statistical analysis
2. Researcher degrees of freedom
3. Weak or open-ended substantive theory
4. Inaccurate summaries of the literature
5. Institutional support
6. External promotion
7. Celebrity culture.

You don’t need all these for a deepfake. In particular, you don’t need bad statistical analysis if you take the trouble to create fake data that shows what you want to see, and you don’t need celebrity culture if you can get your message out in some other way. The real point here is that the deepfake occurs within a social environment.

In an era where we’re worried about algorithmic deepfake science from chatbots, it could be helpful to think about what we’ve learned from algorithmic deepfake science of the human variety.

Blokjutsu: a macOS image generator/editor

This is another post from Bob that is not about statistics. Last one, I promise.

Like Steve, whose what’s up in NYC app I just blogged about, I’ve taken up the hobby of designing apps and having LLMs build them. It scratches a very different itch than writing performant C++ code for Stan.

Blokjutsu

Here’s what I’ve come up with.

  • Blokjutsu: An open-source native macOS app for generating and editing images locally.

Fair warning: it requires Apple Silicon with at least 32GB RAM. On the plus side, it’s distributed as an Apple-notarized .dmg file and licensed entirely under Apache 2, including the models.

MOSS-Image, the 5 billion parameter model on which it is based, is amazing in its ability to follow text instructions. Having said that, don’t expect it to compete with a frontier model in depth or breadth of knowledge. It also generates images up to 4 megapixels, though that can be slow (20 minutes on my old Mac with an M3 Ultra, 12 on my new MacBook with an M5 Max—they’ve accelerated matrix multiplies!).

What does it do?

The app lets you either work entirely from a prompt, or you can structure your input with bounding boxes that are followed very well for either images or text (though the text is handled with fonts and doesn’t integrate real well with photographic images in my opinion). It uses a language model (Qwen) to process prompts and it can be used to enhance prompts directly or to convert images to text. It includes all the usual image-processing features: image to image, variations of generated images, inpainting (replacing part of an image) and outpainting (extending an image). It lets you plug in LoRAs (low rank adaptations) for styling.

I tried very hard to keep everything simple. If anyone has any feedback on either the doc or the system itself, I’d love to hear it, either via email ([email protected]) or an issue on GitHub.

What does “Blokjutsu” mean?

I made up a word that’s pronounceable, has zero Google hits, and isn’t spell corrected (as Hadley Wickham and I were urging instead of “Stan” for the Stan project). It’s a mashup of Danish “Blok” and Japanese “jutsu”. Same for the GitHub org, which is Danish “Form” mashed with Japanese “do”.

Why?

I built this app in part to see what Claude (Opus 5.5, medium effort) could do, and partly because I subscribe to Claude, which doesn’t do image generation! I like to use images in my other hobby, role-playing games. I was using DiffusionBee with StableDiffusion, but DiffusionBee hasn’t been updated in years, and I wanted to be able to use MOSS-Image. The next section has an example.

The Temple of Shadow

My childhood friends were getting tired of my obsession with fiction-first games (like Blades in the Dark) and wanted a “beer and pretzels” (as this style is known) dungeon crawl. I ran “Caverns of Thracia,” which the grognards describe as among the handful of best modules ever written, but we all found it challenging, and quit after the second TPK (total party kill). Hence the 5th edition module, where it’s a real challenge to kill even one character.


Dungeon Module Z1: The Temple of Shadow

The module text was entirely written by ChatGPT, though I provided a lot of editorial oversight, including a suggestion of how to arrange the four broad zones of the dungeon and asking continually for more concise descriptions with more showing and less telling. Everything else, including the serviceable but rather boring cookie-cutter town, was pure GPT. The LaTeX styling was me with GPT’s help. The maps were random generators available online. The images were Stable Diffusion after a lot of metaphorical hair pulling (literal is challenging for me). It’s all a cliché, just as my players requested. The writing is the usual AI type slop and the images are pretty atrocious. If you look at the writing from Wizards of the Coast, I’d call that “WotC slop,” so the bar is low. It’s hard to find good writing in an RPG, so it really stands out reading, for example, a short module by Will Doyle. I’ve been meaning to give this another go in late 2026 and see how Claude does, but I really don’t like reading AI writing.

What I like

Left to my own devices, I run games like I’m running now, Star Trek: Omega, where the Federation is down to its last Class-D starship pulled out of mothballs, with a skeleton crew of officers and enlisted folks who somehow missed the decisive battle in which the Federation was defeated. Captain K’vok (Klingon) and his motley crew are up against the Chorus, an alien computer intelligence harvesting all the planets in the galaxy for compute centers to achieve the singularity. Their five year mission is to make sure the last embers of humanity don’t burn out. I’m reluctant to link it as I don’t want to get a take-down notice, but you can find the Forged-in-the-Dark rules for it on my own GitHub org, bob-carpenter, as well as the rules for Ready, Steady, Go!, my Fast-and-Furious knockoff set in the 1968 Paris student riots, where the English team races against French, Italian, and German competition (all with their iconic sports cars) for the racing title, Le Roi de Paris. The real prize, of course, is much much bigger, and may be uncovered for the first time since the war. If you’re seriously thinking about playing, I can send along the maps and car tokens I used for Roll20. As a taste, here’s the token I generated for one of my players. I could not for the life of me get GPT to render it with right-side drive—I’m hoping MOSS-Image will be better.

The other player’s character drove an Aston Martin Zagato in British racing green, but of course they could pull out the Land Rover for the chase scene through Père Lachaise.

2026 election conspiracies haven’t breached the Vermeule threshold.

Longtime readers of this blog will recall the concept of the John Yoo line: The point at which nothing you write gets taken seriously, and so you might as well become a hack because you have no scholarly reputation remaining.

Speaking of legal scholars who’ve gone off the reservation, I was recently thinking about a related concept, the Vermeule threshold: The level at which an outrageous conspiracy theory becomes acceptable for respectable suit-and-tie types to promote.

This idea is named after the Harvard law professor Adrian Vermeule, who notoriously promoted stupid 2020 election denial material:

This takes on a particularly chilling note given that, just a few years earlier, Vermeule had written:

Because those who hold conspiracy theories typically suffer from a “crippled epistemology,” in accordance with which it is rational to hold such theories, the best response consists in cognitive infiltration of extremist groups.

In case you’re wondering, by “cognitive infiltration,” he explicitly states that he wants the government to be doing this infiltration. So, yeah, this is a very well-connected guy who’s pushing the government to infiltrate political groups—when he’s not promoting stupid-ass conspiracies himself.

I was thinking about this recently because someone in comments pointed to this risible bit of election denial from 2026:

I went online and I couldn’t see any promotion by Vermeule of this or any other 2026 election conspiracy theories. That’s why I characterize these theories as below the Vermeule threshold: so stupid that even the distinguished legal scholar cannot bring himself to endorse them.

Do I think Vermeule was sincere in his promotion of election denial in 2020? No, I doubt it. He’s well educated and, I assume, no idiot. Rather, my guess is that he thought that by circulating such theories he was being a charming imp, offering a good poke in the eye to complacent liberals etc. What a charming rogue, etc. Seeing video of the 6 Jan 2021 insurrection and the government shooting protesters on the street and then lying about it, I consider elite promotion of stupid conspiracy theories to be dangerous, not charming—but maybe part of this is a different perspective coming from a political scientist and a lawyer. I’m trying to understand reality here, with all the difficulties that entails; Vermeule is an attorney and a teacher of law, and that comes with a tradition of advocacy.

So, just to clarify, I don’t think that “below the Vermeule threshold” represents theories that are too stupid to for him to believe; rather, they are too stupid for him to publicly promote. These are theories that are so dumb that the Harvard professor would take more of a reputational hit by endorsing, than he would gain in political points by committing to.

Can you imagine being a law student and taking a class from this guy? This isn’t a case of a researcher with a side interest in extreme political views, along the lines of a J. B. S. Haldane or William Shockley, where you can value the science and engineering and set the politics aside. As with John Dewey, say, Vermeule’s political activism is close to his intellectual work. To be studying from someone who impishly promotes stupid and dangerous conspiracy theories . . . that’s a kind of instrumental view of the law that doesn’t make me happy. I guess that many of Vermeule’s students share his political views, and perhaps the others value the experience of seeing, from close up, the ideology of some of our political rulers. And the Harvard law students surely have the opportunity of experiencing some of this from the other side from their faculty who espouse extreme left-wing views.

P.S. In comments I laid out the argument more carefully:

There is a logic to these spurious election-denial conspiracy theories. It goes like this:

1. The election did not go the way I hoped and expected.

2. Therefore, something wrong happened.

3. I can’t accept that the “something wrong” could be that the majority of the voters legitimately supported the opposition.

4. Therefore, some version of cheating happened.

The next step is for people to make up evidence supporting the claim of cheating, or to suspend one’s critical faculties so as to accept spurious evidence. That’s what Vermeule was doing in 2020, except that I’m guessing he was one step “meta” from that: rather than suspending his critical faculties so as to believe stupid election-denial claims, he thinks he’s being a charming imp by promoting claims that he’s pretty sure are false, as a poke in the eye to the sanctimonious liberals who he sees as a danger to society. The “meta” bit is that I think he views liberals and Democrats as routinely cheating, misrepresenting the arguments of conservatives, abusing government power, etc., so it’s only fair play for this to be done on the other side. Whether or not Vermeule personally comes up with fake election denial theories, he appreciates the efforts of those on his side who do so, in the same sense that a “clean” hockey player respects the absolutely necessary role played by the “enforcers” on his team. No unilateral disarmament, right?

The awkward thing for Vermeule is that he can’t lay out the above logic. When he promotes stupid theories that he surely doesn’t believe, he can’t publicly say that the theories are stupid, or that for political reasons hadn’t looked into them so that he can maintain a plausible deniability about them. He has to play it straight, to follow the rules of the game.

From an intellectual sense, there’s a sort of tragedy here. A smart person who has devoted his intellectual life to the pursuit of logic for political goals, is driven by the logic of politics to endorse ridiculous claims, and his curse is that he cannot allow himself to explain it. He’s very much in the position of those left-wing western intellectuals from the Cold War period who endorsed Stalinism in all its aspects and could never let the mask slip, sacrificing their reputation and their intellectual integrity.

If I weren’t so bothered by the whole insurrection thing, I might feel kind of sorry for the guy.

P.P.S. We had some discussions in a thread from a few months ago about the extent to which Vermeule believes some of the things he says. Here’s one thing I wrote:

One thing that bothers me is that it seems that Vermeule’s extremism is in some ways a professional asset. If he were just a reasonable conservative (e.g., recognizing that Biden won the 2020 election but opposing Biden’s policies, saying that he’s personally religious without saying stupid things about atheists, etc.), then I’m guessing he wouldn’t have attained his current level of fame. I wouldn’t be writing about him, for one thing! Similarly, if Vermeule’s edgelord colleague Cass Sunstein were just a reasonable technocratic liberal and not a prolific streamer of bullshit, he wouldn’t be so famous either. Lots of Ivy League professors write vapid airport nonfiction; Sunstein gets our attention via his over-the-top stupidity.

I suspect that for Vermeule, Sunstein, and other edgelords of their ilk, it is kind of a game—make extreme statements, get publicity—but then they start to believe their own hype. Also, remember, the reasonable positions have already been staked out so there’s a special reward for saying something so stupid that no reasonable person would say it.

Zombie citations: A short empirical investigation

So, we had this discussion the other day about nonexistent or irrelevant citations included in articles produced by chatbots. This is part of a larger concern about AI slop polluting the scholarly literature.

The usual way this is framed is that you look at the reference list of published article or set of articles and then you ask how many of these citations are erroneous in some way.

But maybe it’s easier to look at things the other way: Start with an article, see where it’s been cited, and follow up to see if these citations make sense.

I thought about this in the context of my classic article, published in 2010 on Arxiv and included in a book four years later, “How many zombies do you know?” Using indirect survey methods to measure alien attacks and outbreaks of the undead.

Look up this article on Google Scholar and you’ll see 6 citations. Only 6! That makes me sad. In any case, here they are:

I clicked through to read how my paper was references, and 5 of the 6 basically got it right. The sixth one messes up–it cites the zombies paper as a source for my claim that the probability is about 1 in 10 million that a vote will be decisive–I’m guessing there was some sloppy copying and pasting from my list of published papers.

So, from this short study, my estimate of the probability of a bad citation is 1/6. This was from the pre-AI era, so it’s not a “hallucination”; it’s just someone who was not careful in citation.

If the Jensen Huang / Nvidia spy scandal had happened two weeks ago . . .

If the Jensen Huang / Nvidia spy scandal had happened two weeks ago, I expect the midterm elections would’ve gone pretty much just as they actually went, but lots of pundits would’ve seized on this as the explanation for the election outcomes.

Why do I say this?

Because the result of the election was surprising. Not in its direction—everyone was expecting the Democrats to win a majority of the vote—but in the extent of the majority (in the event, they won 57% of the two-party vote for the House of Representatives) and the consequences for both houses of the legislature.

When there’s a surprise, people look for an explanation. That’s what stories are all about. As the most convenient recent politically-related news item, the Huang scandal would’ve fit the bill.

I’d still guess that, had this all happened fifteen or thirty years ago, the Huang-equivalents would’ve been Democrats. But, as it is, Huang is closely connected to an unpopular Republican president, so the story would’ve written itself. Even though I think that, had the Huang/Huang thing come up two weeks ago, the election outcome would’ve been essentially unaffected.

When is a class a “commodity” and when does it depend on the instructor?

Sometimes I get emails from students asking to work on a project with me, and I usually respond that they should take one of my classes first. If they try to work on a project without having taken a class first, I usually end up having to teach them a bunch of things. Sure, they could read my books, but I guess people don’t really like to read, also there’s something extra you get from hearing it live, doing the homework, getting feedback in class, etc.

This always puzzled me, that students would think they could jump in a project without taking any of my classes first . . . but then I realized that it’s usual to think of a math class as being pretty much the same no matter who teaches it. You take calculus or linear algebra or real analysis or whatever, and it can be taught well or poorly, there are good teachers and bad teachers, but they’re pretty much all teaching the same thing.

But statistics is different. Even if you limit yourself to a narrow topic–for example, regression analysis or survey sampling or Bayesian statistics (just to name three fairly generic-sounding classes that I’ve taught)–the course varies a huge amount from instructor to instructor. Not just teaching style (cos, yeah, my style is much different from that of every other statistics teacher I’ve seen. I’m not saying it’s better, mind you, just that it’s different. You can get a sense of it by reading Part 1 of Active Statistics), but actually what gets taught. First off, most people don’t teach regression out of Regression and Other Stories, and other regression textbooks don’t look much like ours. Second, somehow what gets covered in class isn’t just what’s in the book. This is even true with Bayesian statistics. Lots of people teach out of Bayesian Data Analysis–and I hope that, soon, lots of people will be teaching from Bayesian Workflow–but there are still so many different directions to go.

And I’m not saying I’m unique here. In general, if you want to work with statistician X, it’s probably a good idea to first take a course from X, for just about any X.

In that way, a statistics class is less like a math class or a physics class or a chemistry class and more like a history class or a literature class. Even if the books are set ahead of time, different teachers teach it so differently that they’re just different courses entirely.

But I could see how many students, whose experience is taking straight math courses in high school and college, and maybe some very empty course such as AP statistics or the college equivalent, would look at the roster of statistics classes at a university and just see a collection of topics rather than what it really is, a collection of topics interacted with instructors. And our course descriptions usually don’t do such a good job of conveying that either. Hence this post.

P.S. I’m not saying that a course being a standardized commodity is a bad thing. In a way, that what my colleagues and I have been trying to do with our textbooks, as well as with Active Statistics: we’re trying to give other people the tools to teach as we do. But, the way things stand, it still matters a lot who teaches your class–maybe not such a bad thing, as the students should get some benefit from their big tuition payments. It just seems like a secret to many students that the instructor matters, not just in how well the topic is taught but also in the content of the course.

Just Bronder it: Steve’s bronder.nyc guide to NYC off(-off)-Broadway, jazz, readings, and galleries

This post is from Bob and it is not about statistics.

Steve Bronder, our primary C++ developer for Stan these days, has built a really neat web app (nothing to download/install) consolidating all the fun stuff happening in NYC. It’s absolutely free or I wouldn’t be advertising here.

bronder.nyc: Off-Broadway and off-off-Broadway theatre, readings, jazz, and galleries in NYC right now

The web app is beautifully simple and incredibly useful. Looking for jazz shows, Mitzi and I used to surf the individual venue pages we knew, but the waters are choppy. It’s incredibly slow and frustrating given how badly most of them are laid out for information discovery. Steve’s app just lists everything out in one scrollable list that’s very easy to narrow down to a particular day or interval. The map makes it very easy to find something in walking distance. I didn’t realize just how much I needed this app until he built it and I started using it.

I would highly recommend this if you’re visiting NYC and recommend it even more if you live here. The map mode is super useful for planning things like dates. I’m not just the shill, I’m also a client. Mitzi and I just went on a double date (!) with Nawaf and Christina to the Flatiron Room in Murray Hill this past Sunday night, which was awesome. I found The Flatiron Room on the app, having never heard of the venue before. The Moses Patrou Duo cranked out some really swinging and boppy soul jazz/funk on Hammond organ and drums. Loud enough to hear, but not so loud you couldn’t have a conversation. Putting in a plug for Flatiron Room, there’s no cover, and the Japanese food was really good and reasonably priced, which is about a hundred times better than it needs to be and half the price it could be in a venue like this (this is also true of my fave sushi in NYC, Studio 151, which is above Nublu in the East Village, and has the next tier up of sushi, but still at relatively bargain prices).

P.S. I was amused by Steve’s cookies message, which is a very Steve way of stating the issue: I use Google Analytics to see which shows and theaters are popular. No personal data, no ads. [Gosh I love cookies (Accept)] [Gosh I hate cookies (Reject)].

P.P.S. I am hoping “Bronder” becomes a common word like “Zagat,” but as a verb, like “Google.” I just learned about LLM poisoning attacks, so consider this document 1 of 250.

Bayesian Workflow (my talk in NYC, Wed 18 Nov 2:30pm)

This talk is sponsored by the New York City Chapter of the American Statistical Association, and it’s at CUNY Graduate Center, 365 Fifth Avenue, room 8301 & 8304 (8th Floor).  If you want to attend (in person or remotely; I recommend in person!), just go to the form before 15 Nov and fill it out.

Also here’s a zoom link so if you’re attending remotely, maybe you don’t need to sign up.

Here’s the summary:

Bayesian statistics and statistical practice have evolved over the years, driven by advancements in theory, methods, and computational tools. This book explores the intricate workflows of applied Bayesian statistics, aiming to uncover the tacit knowledge often overlooked in published papers and textbooks. By systematizing the process of Bayesian model development, the book seeks to improve applied analyses and inspire future innovations in theory, methods, and software. It emphasizes the importance of iterative model building, model checking, computational troubleshooting, and simulated-data experimentation, offering a comprehensive perspective on statistical analysis.

Through detailed examples and practical guidance, the book bridges the gap between theory and application, empowering practitioners and researchers to navigate the complexities of Bayesian inference. It is not a checklist or cookbook but a flexible framework for understanding and resolving challenges in statistical modeling and decision making under uncertainty.

The key innovations of Bayesian Workflow include:

    • Going beyond Bayesian data analysis (model building, inference, model checking, and model expansion) to consider the larger process of statistical modeling, including comparisons of multiple models fit to a single dataset.
    • A fuller use of informative priors.  One reason we’ve moved toward stronger priors is that the replication crisis has taught us that the amount of prior information available in any given problem is often approximately the same as the information coming from an experiment (see here, for example). Informative priors also fit our increased focus on generative modeling, and we’re doing a lot more prior predictive checking to understand the implications of our models.
    • More integration between modeling, data analysis, and computing. One way to see this is that the Bayesian Workflow webpage has the code to run all our examples.
    • Simulation-based experimentation.  At its most basic, simulation-based experimentation provides a best-case baseline for statistical methods: if you can’t recover your quantities of interest with sufficient accuracy under ideal conditions (when your data are simulated from the model you’re fitting), then you know you’re in trouble. And often this is the case! Beyond that, we can simulate from one model and fit another, and see what happens. Simulation experiments aren’t always so easy to construct, as they involve specifying the entire data-generation process.

All the information on the book, including the book itself, is at the Bayesian Workflow webpage.

The non-Bayesians among you might be interested in Appendix A of the book, on Statistical and computational workflow for Bayesians and non-Bayesians, and also our short general article, Statistical Workflow, and its accompanying journal issue.

In the talk I’ll follow my usual practice and tell lots of stories.  Stories are how we learn about the world, and I’ll try to also go meta and discuss the anomalous and immutable aspects of the stories I tell.

If you can make it to the talk, please come prepared with some tough questions!

Constructing a latent-variable model to fit measurement data: A polychoric story

Following up on our recent post, Mathias Berggren writes:

I coded up a polychoric estimate for the problem you present in Constructing a latent-variable model to fit measurement data: A Stan story, part 2. I think it led to pretty decent estimates. I attach the R-code in case you want to have a look.

I hadn’t thought about it, but I guess it makes sense that this sort of model is used enough that people would have already hard-coded it!

Here’s the R code:

# Polychoric solution of latent variable stan example from Andrew

# Load required libraries
library(polycor)

# Data as a matrix
data = matrix(c(208,501,51,640,9879,1439,40,464,1432), nrow = 3, ncol = 3)

# Observed joint probabilities
Pobs = data / sum(colSums(data))

# Polychoric estimate of the latent correlation
r = polychor(data)

# Observed marginal probabilities
pX = colSums(data) / sum(colSums(data))
pY = rowSums(data) / sum(colSums(data))

# Corresponding marginal thresholds
cX = c(qnorm(pX[1]),qnorm(pX[1]+pX[2]))
cY = c(qnorm(pY[1]),qnorm(pY[1]+pY[2]))

# Estimated joint probabilities from this information
P = matrix(nrow = 3, ncol = 3)
P[1,1] = integrate(function(y) dnorm(y) * pnorm((cX[1]-r*y)/sqrt(1-r^2)), -Inf, cY[1])$value
P[1,2] = integrate(function(y) dnorm(y) * (pnorm((cX[2]-r*y)/sqrt(1-r^2)) - pnorm((cX[1]-r*y)/sqrt(1-r^2))), -Inf, cY[1])$value
P[2,1] = integrate(function(y) dnorm(y) * pnorm((cX[1]-r*y)/sqrt(1-r^2)), cY[1], cY[2])$value
P[2,2] = integrate(function(y) dnorm(y) * (pnorm((cX[2]-r*y)/sqrt(1-r^2)) - pnorm((cX[1]-r*y)/sqrt(1-r^2))), cY[1], cY[2])$value
P[1,3] = pY[1] - P[1,1] - P[1,2]
P[2,3] = pY[2] - P[2,1] - P[2,2]
P[3,1] = pX[1] - P[1,1] - P[2,1]
P[3,2] = pX[2] - P[1,2] - P[2,2]
P[3,3] = pX[3] - P[1,3] - P[2,3]

# Compare estimated and observed probabilities
P
Pobs
Pobs - P

# Find least squares estimates to transform thresholds to 1.5 and 3.5
b1X = sd(c(1.5, 3.5)) / sd(cX)
b0X = mean(c(1.5, 3.5)) - b1X * mean(cX)

b1Y = sd(c(1.5, 3.5)) / sd(cY)
b0Y = mean(c(1.5, 3.5)) - b1Y * mean(cY)

# Check  so we get the correct values
b0X + b1X * cX
b0Y + b1Y * cY

# Least squares estimates
# These show how we should transform the standard normal marginal 
# distributions above to a scale with thresholds 1.5 and 3.5.
# Thus, they also indicate the updated mean and standard deviation of 
# each variable (as they transform from one with mean = 0 and sd = 1).
# If one wants one common mean, then one might take some weighted mean 
# of the intercepts
c(b0X,b1X)
c(b0Y,b1Y)

# Discussion
# If there are more thresholds then the least squares solutions may not be 
# exact, and we may find that the thresholds cannot be at 1.5, 2.5, etc.
# Same way, one might find that the intercept cannot be the same.

The challenge of truly probabilistic programming

Many years ago, Jouni and I wrote an unpublished paper called Fully Bayesian Computing:

A fully Bayesian computing environment calls for the possibility of defining vector and array objects that may contain both random and deterministic quantities, and syntax rules that allow treating these objects much like any variables or numeric arrays. Working within the statistical package R, we introduce a new object-oriented framework based on a new random variable data type that is implicitly represented by simulations.

A few years later we published an updated version of the paper under the title, Manipulating and summarizing posterior simulations using random variable objects. Here’s the background:

In the Bayesian paradigm, unknown quantities have probability distributions and are thus random variables. Observed values are just realizations of random variables, and constants may be thought of as random variables with point mass distributions. In mathematical notation, we deal with objects that are random variables, but in practice these objects are approximated by vectors of numbers, that is, simulations. Consequently, when programming for manipulating simulations of unknown quantities, we must write code to manipulate arrays of numeric constants.

And here’s the challenge:

Arrays of simulations are cumbersome objects to work with. Functions that work with vectors will not in general work with matrices, so special versions of the functions need to be written to accommodate matrices of simulations as arguments. For example, a scalar-valued random variable becomes a vector of simulations, and a random vector becomes a matrix of simulations. . . . The usual approach now is to use loops and vector-matrix computations to manipulate the matrix of posterior simulations. This approach is general but awkward and far from transparent . . .

As part of this effort, Jouni wrote a package in R called “rv” that allows direct manipulation of random variable objects, so for example if alpha and beta are random variables, each a vector of length J, and gamma is a scalar, then you can compute something like delta = (alpha + gamma)/beta, which would be a vector of length J. Internally, alpha, beta, and delta would be stored as S x K arrays, and gamma would be stored as an array of length S, where S is the number of simulation draws in your simulation frame, but the user would be able to work with these without having to keep track of that indexing; it would all be implicit.

The rv package was just a demonstration, but then more recently Paul Bürkner, Jonah Gabry, Matthew Kay, and Aki Vehtari implemented the idea in the “posterior” package in R. They call it rvar: The Random Variable Datatype. I still haven’t integrated rvar() into my Bayesian workflow, but I’d like to do it.

The point I want to make here is that this random variable datatype moves us closer to what I’d call “truly probabilistic programming.”

Here’s what I mean by that.

Stan (and Bugs, PyMC, etc.) are called “probabilistic programming languages,” characterized by Wikipedia as “based on the declarative specification of probabilistic models, for which inference is performed automatically.” Fair enough. In Stan, the variables defined in the parameters block, transformed parameters block, model block, and generated parameters block represent random variables with a joint density that is specified by the Stan program.

But, although this is called a probabilistic program as the term is used, it doesn’t fully feel like probabilistic programming in the sense of allowing me to grab random variables and work with them directly. I can do all that within the Stan program in the generated quantities block, but at a certain remove. Stan and other probabilistic programming languages are set up to run in batch mode: you specify a model and then the inference engine runs and spits out posterior simulations (or some other distributional summary).

To me, a truly probabilistic programming environment would allow me to interactively work with random variables, and to work with them as random variables, not as arrays of simulation draws. I want to be able to multiply two matrices without having to loop through the simulations. Part of this is about clean code, but more generally it’s about workflow. Any mathematical or graphical operations we can do on numbers, we should be able to do with random variables, with the simulations all hiding under the hood. Kind of by analogy to how we can do floating-point math on the computer without (usually) having to worry about how all these digits are being stored.

This will require tools such as rvar() and also some experience with the workflow. The payoff is . . . you could call it “better postprocessing of Bayesian simulations,” but I think that would understate the potential benefits of being able to work directl with uncertainty without that extra array dimension getting in your face at every step.