How accurate are AI text detectors? What the peer-reviewed studies measured

What this review covers

"How accurate are AI detectors?" is usually answered with a vendor's number. This review answers it with the independent literature instead: the largest multi-tool academic evaluation (Weber-Wulff et al., 14 tools), the one case where a vendor published its own accuracy and then retired the product over it (OpenAI), a 2024 adversarial trial in a higher-education setting (Perkins et al.), the largest shared benchmark in the field (RAID, over six million generations), and the theory paper that explains why the numbers look the way they do (Sadasivan et al.).

Selection rule: studies had to be independent of the vendors being measured (OpenAI's self-report is included precisely because it is a vendor testifying against its own product), had to report their method and numbers, and had to be retrievable in full. Every quotation is verbatim from the linked source, checked on the asOf date. One sourcing caveat is flagged inline: OpenAI's original classifier pages are no longer reachable, so its figures are cited through contemporaneous press that quoted them.

The pattern across all of them: moderate accuracy on clean, unmodified AI text; sharp collapse under paraphrase, translation, or editing; and false positives that never reach zero.

Weber-Wulff et al. (2023): fourteen tools, none reliable

The reference study for education is "Testing of detection tools for AI-generated text" by Weber-Wulff, Anohina-Naumeca, Bjelobaba, Foltýnek, Guerrero-Dib, Popoola, Šigut, and Waddington — a cross-European team of academic-integrity researchers — published in the International Journal for Educational Integrity (19:26, December 2023).

They tested fourteen systems: Check For AI, Compilatio, Content at Scale, Crossplag, DetectGPT, Go Winston, GPT Zero, GPT-2 Output Detector Demo, OpenAI Text Classifier, PlagiarismCheck, Turnitin, Writeful GPT Detector, Writer, and Zero GPT — "12 publicly available tools and two commercial systems (Turnitin and PlagiarismCheck)". Test documents covered six conditions: human-written, human-written-then-machine-translated, AI-generated, AI-generated in a second batch, AI-generated with human editing, and AI-generated with machine paraphrasing.

The topline, verbatim:

"Detection tools for AI-generated text do fail, they are neither accurate nor reliable (all scored below 80% of accuracy and only 5 over 70%)."

The direction of the errors matters. The study found the tools "have a main bias towards classifying the output as human-written rather than detecting AI-generated text" — that is, the dominant failure is missing AI text. But false positives occur too, and the authors quantify both:

"In general, they have been found to diagnose human-written documents as AI-generated (false positives) and often diagnose AI-generated texts as human-written (false negatives)."

The per-condition numbers show exactly where accuracy lives and dies:

• Human-written text: 96% accuracy. This is the honest-student case, and it is the best case in the study.
• Human text machine-translated into English: accuracy dropped by 20%. The same human writing, passed through a translator, starts triggering detectors — the failure mode behind the non-native-speaker findings covered in our bias review.
• Unmodified AI text: 74%. One in four AI-written documents sailed through even with no evasion effort at all.
• AI text with human editing: 42% — "slightly over half the score" of the unmodified cases. Light human revision halves detection.
• AI text machine-paraphrased: 26%. Verbatim: "most AI-generated texts remain undetected when machine-paraphrased."

On false accusations specifically:

"Six of the fourteen tools tested generated false positives… For GPT Zero, half of the positive classifications would be false accusations, which makes this tool unsuitable for the academic environment."

And the authors' conclusion, in full:

"Moreover, it is too easy to game the systems by using paraphrasing tools or machine translation. Therefore, our conclusion is that the systems we tested should not be used in academic settings."

That sentence is from the peer-reviewed literature, not from a provenance vendor's marketing. It is the single most-cited finding in this field.

OpenAI (2023): the vendor that published its own accuracy

OpenAI launched an "AI Text Classifier" in January 2023 and retired it six months later. Its retirement note — quoted by The Register on 26 July 2023, since the original page is no longer online — read:

"As of July 20, 2023, the AI classifier is no longer available due to its low rate of accuracy."

What did "low" mean? At launch, OpenAI's own published evaluation — quoted contemporaneously by Synthedia from OpenAI's announcement — said the classifier:

"correctly identifies 26% of AI-written text (true positives) as 'likely AI-written,' while incorrectly labeling human-written text as AI-written 9% of the time (false positives)."

Read those two numbers together. The tool caught about one in four AI texts, and accused roughly one in eleven human writers. The company with the most knowledge of its own models' output — and the strongest commercial incentive to make detection work — measured this, published it, shipped anyway, and then withdrew the product. (Sourcing caveat: both quotations come via press coverage of OpenAI's pages, which were verified unreachable at the primary URL as of this review's asOf date.)

Perkins et al. (2024): six detectors under realistic evasion

Perkins, Roe, Vu, Postma, Hickerson, McGaughran, and Khuat tested "six major Generative AI (GenAI) text detectors" against 805 text samples in a higher-education context, including samples modified with simple adversarial techniques a student could apply. From the abstract, verbatim:

"the detectors' already low accuracy rates (39.5%) show major reductions in accuracy (17.4%) when faced with manipulated content"

Two readings of this result, both uncomfortable. First, the baseline — before any evasion — was 39.5% in this study's setting, far below any vendor-published figure. Second, trivial manipulation cut that more than half, to 17.4%. The authors' conclusion:

"the potential for false accusations demonstrate that these tools cannot currently be recommended for determining whether violations of academic integrity have occurred."

RAID (ACL 2024): the largest benchmark, and the false-positive problem

The benchmark paper in this set is Dugan et al.'s RAID ("A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors," ACL 2024), built by researchers at Penn and collaborators. Its scale dwarfs everything above:

"RAID includes over 6 million generations spanning 11 models, 8 domains, 11 adversarial attacks and 4 decoding strategies."

They evaluated "8 open- and 4 closed-source detectors." The motivating observation is aimed squarely at vendor claims:

"Many commercial and open-source models claim to detect machine-generated text with extremely high accuracy (99% or more). However, very few of these detectors are evaluated on shared benchmark datasets and even when they are, the datasets used for evaluation are insufficiently challenging..."

The findings are organized bluntly. The paper's first finding header reads: "Default False Positive Rates (FPRs) of open-source detectors are dangerously high." Verbatim from the full text:

"We see that open-source detectors, especially metric-based detectors, exhibit dangerously high false positive rates when using these naive thresholds."

To compare tools fairly, the authors re-thresholded every detector to a fixed 5% false-positive rate — and found that "Detector accuracy varies substantially depending on target False Positive Rate." A detector that looks excellent at a lenient threshold can be "nearly inoperable" at a strict one. This is the technical reason a vendor's single accuracy number tells you almost nothing: without the false-positive rate it was measured at, the number is not interpretable.

The paper's overall conclusion, verbatim:

"Detectors are not yet robust enough for widespread deployment or high-stakes use: many detectors we tested are nearly inoperable at low false positive rates, fail to generalize to alternative decoding strategies or repetition penalties, show clear bias towards certain models and domains, and quickly degrade with simple black-box adversarial attacks."

RAID is not uniformly bleak — the authors note bright spots: "Binoculars, for example, performed impressively well across models even at extremely low false positive rates, Originality achieved high precision in some constrained scenarios, and GPTZero was unusually robust to adversarial attacks." But even so: "Our results also support this and suggest that the problem of false positives remains unsolved."

Sadasivan et al.: why the ceiling exists

The theory paper that frames all the empirical results is "Can AI-Generated Text be Reliably Detected?" by Sadasivan, Kumar, Balasubramanian, Wang, and Feizi (University of Maryland), published in TMLR. Its empirical contribution is the recursive-paraphrasing attack covered in detail in our paraphrase-attacks review; its theoretical contribution bounds what any detector can ever achieve:

"we provide a theoretical framework connecting the AUROC of the best possible detector to the Total Variation distance between human and AI text distributions."

Translation: detection accuracy is capped by how different human and AI text actually are, as statistical distributions. As models improve, that distance shrinks — and the best possible detector's performance shrinks with it, approaching a coin flip in the limit. The paper also demonstrates the inverse attack, spoofing: "an attacker can infer hidden AI text signatures without white-box access to the detection method" — meaning a bad actor can make human text look AI-generated, "potentially leading to reputational risks for LLM developers" and, in a classroom, to manufactured evidence against an innocent writer.

The studies, side by side

Study	Scale	Clean-AI accuracy	Under evasion	False-positive finding
Weber-Wulff et al. 2023	14 tools, 6 conditions	74%	42% (edited), 26% (paraphrased)	6 of 14 tools produced false positives; GPT Zero: half of positives would be false accusations
OpenAI self-report 2023	own classifier	26% true positives	—	9% of human text flagged; retired for "low rate of accuracy"
Perkins et al. 2024	6 tools, n=805	39.5%	17.4%	"potential for false accusations… cannot currently be recommended"
RAID (Dugan et al.) 2024	12 detectors, 6M+ generations	varies by threshold	"quickly degrade with simple black-box adversarial attacks"	default FPRs "dangerously high"; "the problem of false positives remains unsolved"
Sadasivan et al. (TMLR)	theory + attacks	—	recursive paraphrase defeats all families tested	spoofing can frame human text as AI

What the literature does and does not establish

Established:

• No independently evaluated detector reached 80% overall accuracy in the largest multi-tool academic study, and only five of fourteen cleared 70% (Weber-Wulff et al.).
• Accuracy on unmodified AI text is the best case, and it is already leaky (74% in Weber-Wulff; 39.5% baseline in Perkins' setting). Light editing or machine paraphrase cuts detection to 42% and 26% respectively.
• False positives are persistent across every study that measured them: 9% for OpenAI's own classifier, six of fourteen tools in Weber-Wulff, "dangerously high" default rates in RAID. The peer-reviewed conclusion in the integrity literature is that these tools "should not be used in academic settings."
• A detector's accuracy number is meaningless without the false-positive rate it was measured at (RAID's central methodological point).
• There is a theoretical ceiling: best-possible detection accuracy is bounded by the statistical distance between human and AI text, which shrinks as models improve (Sadasivan et al.).

Not established:

• That detectors are useless as screening signals. Several tools performed respectably in constrained scenarios (RAID's notes on Binoculars, Originality, and GPTZero's adversarial robustness). The literature's claim is narrower and sharper: the error rates are incompatible with using a score as proof in a high-stakes accusation.
• Current accuracy against current models. Every study here evaluated the detectors and generators of its moment; both have changed since. The structural findings (threshold sensitivity, paraphrase fragility, the TV-distance bound) are the durable part, not any specific percentage.
• Per-student error rates in real classrooms. These are corpus studies; none of them measures how often a specific honest student gets flagged over a semester of real submissions.

Why this matters for provenance

Every number above describes an attempt to infer, from finished text alone, how that text was made. The literature's verdict is that this inference is unreliable in both directions — and the direction that ruins lives is the false positive, which no study managed to eliminate. That is the case for evidence that does not require inference: a record of the writing process, made while the writing happened, that an honest writer can simply present. See what a Turnitin AI score actually means, what to do if you are falsely accused, and what writing provenance is.

Related reading: non-native speakers and detector bias · paraphrase attacks on AI detectors · LLM watermarking — where the research stands