This is Jessica. I continue to be amazed by how much confusion there is about Pangram, the AI text detector. As Sam Kriss reports on Substack, not only are people sharing misinformation amongst themselves, the idea that “Pangram is racist” and results in high false positive rates for African or Caribbean or non-native English speakers is worming its way into the record, unbacked by any solid evidence. AI search results are now promoting assertions about bias that are challenged by existing audits of Pangram on texts from non-native English speakers. Academics are calling such tools racism clothed “in a veneer of objectivity.” Motivating such accusations are several cases where POC short story award winners won with texts that Pangram flagged as 100% AI-authored. The result is a dent in our collective understanding of basic binary classification.
With all of this apparent controversy, it’s reasonable to wonder what is going on. Let’s review what is actually true about Pangram’s error rates, and then consider why so many people are buying into the attempts to make this a social justice issue.
But first, if you read no further, the TLDR is that false positives (where Pangram flags human-generated text as AI-generated) on texts of reasonable length (at least 50 words, with greater reliability as length increases) are extremely unlikely according to every general audit I could find – the largest report FPRs on the order of 1 in 10,000 for audits on mixed length texts. In fact, Max Spero, Pangram CEO, posted on X a few days ago that based on a bounty program, they had found their first false positive on a long text created before 2020: a poem that repeats the same six words four thousand times. False positive rates on non-native English speaker corpora remain indistinguishable from these overall rates according to large audits. This doesn’t mean Pangran is reliable on every distribution of human-authored text, adversarially generated or not. But the evidence of low false positives on natural distributions of human texts is much stronger than the cherry-picked data points the racism-accusers are wielding.
Audit results
According to the latest model card published by Pangram itself, on Pangram 4.1 released October 19, 2026, overall false positive rates are:
Note that Pangram uses a strict definition of false positive: if a known-to-be human written document has any segment classified as AI-generated or AI-assisted, the entire document counts as a false positive. Similarly, a fully AI-generated document with any segment labeled human or AI-assisted is counted as a false negative.
Overall false negative rates are higher than false positives (by about 40x) but still quite low:
The above makes clear that error rates are asymmetric. This is intentional, and something a model developer can control, by using a training objective that prioritizes avoiding false positives or more directly by setting a conservative classification threshold learned on held-out data. This makes it possible to ensure a very low FP rate on text generated similarly to the audit distribution (Pangram describes targeting 1 in 10,000, and available audit evidence is consistent with this).
When Emily Bender says that “As is widely known within the NLP research community, perfect results usually point to something wrong with the evaluation setup”, she’s either deliberately ignoring or unaware of basic statistical machine learning methods.
Other, smaller audits of Pangram models include Epoch AI’s from this past summer, which found that Pangram produced zero false positives on 495 human passages. It detected all of 297 AI passages created using basic prompts, but missed 30 of 297 (10.1%) when the LLM was prompted to imitate a particular author’s style. Another by Karr, Khvatskii, Hua, and Chawla from last month finds that Pangram had zero false positives on 306 pre-LLM abstracts (2013-2015), but it flagged 14.9% of recent abstracts (from 2023 – 2025) as AI-generated. However, the authors don’t know what the ground truth was on the recent ones, so this statistic cannot be treated as a false positive rate. A very small audit by Van Vlasselaer, Van Droogenbroeck & Spruyt from June, which found that Pangram detected 97.5% of 40 fully AI-generated papers and about 95% of hybrid human and AI generated papers and humanized AI-generated papers. In one of the first external audits in 2025, Imas and Jabarian found that version 2 Pangram achieved nearly zero false positives and false negatives on medium and long texts.
Where is the evidence that Pangram is racist?
I am not aware of any audits with known ground truth specifically focused on texts by POC, but there are a few related analyses. Vauhini Vara reports that when she ran 25 prior prize winning stories of the Caine Prize for African Writing through Pangram, all came back 100% human-authored.
Adding to this, yesterday Tuhin Chakrabarty posted an analysis of books written by African authors: 97 from Zimbabwean authors written 2020 to 2025 and 90 from other African countries written from 2020 to 2026. He found just one that was flagged at least 1% AI by Pangram.
Beyond the claims that Pangram is racist against African or Caribbean authors, there is a general line of attack that it should not be used because errors rates are unacceptably high for non-native English speakers. Yet Pangram has produced multiple rounds of audits using four non-native english text corpora.
Hardly higher than the overall rates reported above.
An audit of version 4 compared to 3.3 shows lower rates:
And an audit of 4.1 from October this year on 15,420 documents by English-as-a-second-language undergraduate and graduate students produces literally zero false positives:
So why are so many people confused?
Here I can only speculate.
Incentives for organizations to save face. Both the Commonwealth Prize and Caine prize organizations report doing investigations that supported the accused authors’ case. If you’re looking to the literary authorities to tell you what to think, it seems that false positives are occurring repeatedly for long texts by POC authors. However, neither has made any of the evidence they claim to have gathered public, so believing that the stories are false positives means trusting their word.
Misinfo to protect AI habits. Blatant misinformation is certainly being propagated, often by people who appear to want to keep their current productivity edge by having AI write everything for them. Here’s just one example of someone posting a faked Pangram false positive screenshot on X:
Confusing FPs and FNs, confusing different classification tasks. Some may simply be mistaking overall accuracy or false negative rates as evidence that Pangram is not reliable when it flags something as AI-generated. There also seems to be some confusion about what Pangram does – detect who strung the words together – versus other forms of detection we might want, like being able to tell who came up with the ideas. But this is not what Pangram targets – if you use AI to rewrite human texts, like rewriting abstracts in the Karr et al study above, and you don’t take steps to constrain the AI from only fixing typos or other errors, then yes, Pangram will flag it as highly AI-generated, because that is what Pangram it is designed to do.
There are also people arguing against use of Pangram based on error rates when texts are generated adversarially, like examples that were specifically written to sound like AI, in contrast to the audit results above which apply to naturalistic distributions of human-generated text. When we target a particular FPR for a classifier, we expect that to hold only so long as the training or calibration data we used is representative of the cases we encounter in practice.
Finally, how to interpret Pangram percentages can be confusing, and attributions of the source of small pieces of text within a larger text are not covered by the audits above. Unreliability in identifying which portion of a text contributes a high score can suggest which use cases to be careful with, but it doesn’t invalidate the document-level false-positive estimates from the audits.
Splashy earlier results on bias in other detectors. A early (2023) Stanford study by James Zou and others did claim that GPT detectors are biased against nonnative English writers, but this was on a very small sample (91 essays from theTOEFL exam) and did not test Pangram.
Inaccurate folk theories. Another contributor might be inaccurate folk theories about what Pangram is. Maybe some people imagine Pangram doing something like they would do if they were given a text and asked if it was AI-generated – e.g., look for cues that are said to be associated with AI text like m-dashes and decide based on that. There are some stories consistent with this, like where Granta turned to a Claude instance to help them decide if AI had been used rather than a trained detector. In reality, Pangram is a classifier, trained with lots of labeled texts to discriminate between which are human-generated and which are AI-generated, and calibrated to keep the false positives low on average in naturally generated texts.
Resisting AI as a solution to anything. Finally, I think a lot of people resent the imposition of AI in their lives, and see relying on an AI detector like Pangram as accepting the inevitability of AI, which they want to resist. So they say things that imply Pangram is unreliable for all purposes without realizing where the evidence contradicts their assumptions.
Concern that detectors are more often applied to POC and ESL writers. Maybe people are perceiving real asymmetries in whose work is being checked, and reacting to that by trying to discredit the tool.
On the larger question of whether we should care about AI writing
We need to be much more careful in distinguishing between the evidence on false positives and the broader question of whether it’s a good idea to try to detect AI writing. Everyone is entitled to an opinion on the latter, but the former is better left to the statisticians. If you want to avoid using Pangram on the grounds that you believe the correlation between who wrote the words and who controlled the ideas is so low that your decision quality will be no better or worse, then I can respect that – we have different priors. Or if you personally want to avoid developing a habit of relying on AI detectors because you believe that human writing will become more and more resemblant of AI writing with time, or AI-written text less indistinguishable, that’s fine, it’s up to you how you make decisions for yourself.
What I really dislike is the way that people (including academics like Bender) are adding confusion by ignoring (and even actively arguing against) how a positive classification from a classifier like Pangram can have evidential value for some of the decisions people care about right now. Sure, false positives are possible, but this does not warrant recommending to people that they remain in ignorance on every case, even if they might benefit from some signal on who wrote the text. Sure, distributions can shift, but there is a difference between recognizing that estimates may not be calibrated in a new context, and concluding that existing evaluations are “effectively meaningless.” It’s like all these Pangram-haters want to scare everyone else into being as irrational as they are by preying on their fear of being labeled racist.
More on the rationality of using a signal when it is informative here.




















