Are AI detectors biased against non-native English speakers? The evidence

What this review covers

The general accuracy of AI detectors is covered in our detector-accuracy review. This review isolates one specific failure mode with its own literature: the documented tendency of detectors to flag writing by non-native English speakers as AI-generated at drastically higher rates than writing by native speakers.

The primary study is Liang, Yuksekgonul, Mao, Wu, and Zou (Stanford), "GPT detectors are biased against non-native English writers," published in Patterns (4(7), July 2023). Around it sit three corroborating documents: the machine-translation condition in Weber-Wulff et al.'s fourteen-tool study, OpenAI's statement that this risk was one reason it withheld its own text watermark, and Perkins et al.'s framing of detector error as an inclusivity problem. Every quotation is verbatim from the linked source, checked on the asOf date.

Liang et al. (2023): the TOEFL study

The design is simple enough to summarize in one sentence, and that is part of its force. From the published full text:

"we evaluated the performance of seven widely used GPT detectors on 91 TOEFL (Test of English as a Foreign Language) essays from a Chinese forum and 88 US eighth-grade essays from the Hewlett Foundation's ASAP dataset."

Both corpora are human-written. The TOEFL essays predate ChatGPT's release; the ASAP essays are from a standard educational dataset. If detectors were measuring "was this written by a machine," both sets should pass. Instead:

"While the detectors accurately classified the US student essays, they incorrectly labeled more than half of the TOEFL essays as 'AI-generated' (average false-positive rate: 61.3%)."

Three numbers from the paper, each worth stating separately because each describes a different kind of harm:

• 61.3% average false-positive rate on the human-written TOEFL essays, across the seven detectors. (The paper identifies the detectors only in a figure; the text describes them as "seven widely used GPT detectors.")
• 19.8% flagged unanimously — "All detectors unanimously identified 19.8% of the human-written TOEFL essays as AI authored." One in five of these human writers would be flagged no matter which tool their institution happened to buy. Cross-checking tools against each other — a common "safeguard" — would not have saved them.
• 97.8% flagged by at least one detector — "at least one detector flagged 97.8% of TOEFL essays as AI generated." Under a policy of "we run several tools and investigate anything any of them flags," essentially every writer in this corpus gets investigated.

Meanwhile, the figure-1 caption records the asymmetry: "High misclassification of TOEFL essays written by non-native English authors as AI generated, with near-perfect accuracy for US eighth-grade essays." The same tools, the same day, near-perfect on one human population and wrong more often than right on another.

The perverse mitigation

The study's most quoted twist is what fixed the false positives. The authors asked ChatGPT to rewrite the flagged human essays in fancier language:

"upon prompting ChatGPT to self-edit its text with more literary language (prompt: 'Elevate the provided text by employing literary language'), detection rates plummeted to near zero (Figure 2)."

Sit with the logic of that result. Human text written by a non-native speaker is flagged as AI. Pass that same text through an AI, and it stops being flagged. The detectors are not measuring machine involvement — under this experiment they are measuring something closer to linguistic plainness, which correlates with machine involvement in one population and with second-language writing in another. The abstract draws the same double conclusion:

"simple prompting strategies can not only mitigate this bias but also effectively bypass GPT detectors"

So the signal fails both ways at once: it accuses honest plain writers, and it clears dishonest users who spend one extra prompt.

The authors' recommendation

The abstract's closing position, verbatim:

"[we] caution against their use in evaluative or educational settings, particularly when they may inadvertently penalize or exclude non-native English speakers from the global discourse."

Weber-Wulff et al.: the machine-translation corroboration

The fourteen-tool study reviewed in our accuracy review included a condition directly relevant here: human-written documents in languages other than English, machine-translated into English (their case "02-MT"). These are human ideas, human-authored, passed through a translation tool — the everyday workflow of many multilingual students and researchers. Result, verbatim:

"in the case of the documents written by humans in languages other than English that were machine-translated to English (case 02-MT), the accuracy dropped by 20%."

Against a 96% baseline on untranslated human text, translation alone pushed a meaningful share of human work into the flagged zone. Different corpus, different team, different tools than Liang et al. — same direction: text that reaches English through a second-language pathway is systematically more likely to be called AI.

OpenAI: the vendor that named the risk

When OpenAI explained why it had not released its text watermarking system — a system it describes as "highly accurate" — one of the two stated reasons was precisely this population. Verbatim, from "Understanding the source of what we see and hear online":

Text watermarking "could stigmatize use of AI as a useful writing tool for non-native English speakers."

This is worth pausing on because of who is saying it. The vendor best positioned to profit from AI-text identification looked at the educational use case and concluded that the equity risk to non-native speakers was a reason to keep a working system unreleased. (The page also cites false-positive volume at scale as the other reason; see our watermarking review.)

Perkins et al.: detector error as an inclusivity problem

The 2024 higher-education trial by Perkins et al. — six detectors, 805 samples — carries the framing in its title: "GenAI Detection Tools, Adversarial Techniques and Implications for Inclusivity in Higher Education." Its measured accuracy (39.5% baseline, 17.4% under manipulation) is covered in the accuracy review; its conclusion belongs here too, verbatim:

"the potential for false accusations demonstrate that these tools cannot currently be recommended for determining whether violations of academic integrity have occurred."

False accusations are not distributed evenly. The Liang results say who absorbs them.

Why the bias happens

The mechanism is not mysterious, and it follows from how these detectors work (explained at length in how AI detectors work — and why they break). Statistical detectors score text by predictability: how closely each next word matches what a language model would expect. AI text is predictable because a model generated it. But second-language writing is also more predictable on average — vocabulary drawn from a smaller working set, safer and more conventional phrasing, grammar learned from the same textbook patterns a model has seen millions of times. The detector cannot tell these two sources of predictability apart, because on the page they are the same measurement. Liang et al.'s elevation experiment is the clean demonstration: add lexical variety (ironically, via an LLM) and the "AI" verdict evaporates.

This also explains why the bias is a structural property of perplexity-style detection rather than a bug any one vendor can patch: lowering the threshold to protect plain writers lets more actual AI text through; raising sensitivity to catch AI text flags more plain human writers. The trade-off can be moved, not removed.

The evidence, side by side

Source	Population	Finding
Liang et al. 2023 (Patterns)	91 human TOEFL essays vs 88 US 8th-grade essays, 7 detectors	61.3% average FPR on TOEFL essays; 19.8% flagged unanimously; 97.8% flagged by ≥1 detector; "near-perfect accuracy" on the US essays
Liang et al., elevation experiment	same TOEFL essays, AI-rewritten	detection "plummeted to near zero" after literary-language rewrite
Weber-Wulff et al. 2023	human text machine-translated to English, 14 tools	accuracy dropped by 20% vs untranslated human text
OpenAI	its own unreleased watermark	"could stigmatize use of AI as a useful writing tool for non-native English speakers"
Perkins et al. 2024	6 detectors, n=805, higher-ed	"cannot currently be recommended" for integrity determinations, citing false-accusation potential

What the literature does and does not establish

Established:

• On the corpora tested, widely used detectors false-positived on a majority of human essays by non-native English writers (61.3% average) while achieving near-perfect accuracy on native-speaker essays from the same experiment (Liang et al.).
• Consensus among tools does not protect this population: one in five TOEFL essays was flagged by every detector tested.
• The effect reproduces across methodologies: machine-translated human text lost 20 points of accuracy in an entirely independent study (Weber-Wulff et al.).
• The bias is mechanistically linked to predictability scoring — the same rewrite that increases lexical variety eliminates the false positives (Liang et al.'s elevation experiment).
• The concern is acknowledged by a major vendor as a reason to withhold a working detection technology (OpenAI).

Not established:

• Precise flag rates for populations other than the ones tested. The headline numbers come from one corpus of TOEFL essays by writers on a Chinese forum; other language backgrounds, proficiency levels, and genres were not measured in these studies.
• That current commercial detectors show the same magnitudes today. The Liang measurements are from 2023 tools; vendors have adjusted thresholds since. The structural trade-off (plainness vs. sensitivity) is the durable finding; any specific percentage is a snapshot.
• Intent. Nothing in this literature suggests vendors designed for this outcome — it is an emergent property of perplexity-based detection that the studies document and the vendors' own statements acknowledge.

Why this matters for provenance

A non-native speaker flagged by a detector faces a uniquely bad evidentiary position: the accusation rests on the texture of their English, and their honest explanation — "this is just how I write" — is exactly what the detector has already discounted. The literature above says the tool cannot distinguish their plain prose from machine output. A process record can: it shows the hours, the drafts, the revisions, the hesitations — made while the work happened, independent of how the final sentences read. See what to do if you are falsely accused, how to prove you didn't use AI, and what writing provenance is.

Related reading: how accurate are AI text detectors? · paraphrase attacks on AI detectors · LLM watermarking — where the research stands