Do AI detectors live up to their accuracy claims? Vendor numbers vs independent tests

What this review covers

Detection vendors publish numbers: accuracy percentages, false-positive rates, validation corpus sizes. Independent researchers publish different numbers for the same tools. This review puts both sets side by side, quoted verbatim and dated, so the gap — and the reasons for it — can be examined rather than asserted.

Method: the vendor claims below were retrieved from the vendors' own pages (fetch route noted where it was indirect). The independent measurements come from the peer-reviewed integrity literature and the ACL 2024 shared benchmark. One excerpted source is flagged inline. Two fairness notes up front: several vendors publish meaningful caveats alongside their numbers, and those are quoted too; and differences between a vendor's number and an independent one do not by themselves prove bad faith — corpora, thresholds, and definitions differ. What the differences do prove is that the vendor's number cannot be assumed to describe your documents.

The framing finding

The ACL 2024 RAID benchmark states the industry-wide pattern:

"Many commercial and open-source models claim to detect machine-generated text with extremely high accuracy (99% or more). However, very few of these detectors are evaluated on shared benchmark datasets and even when they are, the datasets used for evaluation are insufficiently challenging..."

Keep that in mind as a lens for everything below. (Full close reading: RAID and M4.)

Turnitin: the most carefully hedged claims

Turnitin's figures are the most consequential, because its detector is the one integrated into institutional workflows. Its current FAQ states:

"We strive to maximize the effectiveness of our detector while keeping our false positive rate - incorrectly identifying fully human-written text as AI-generated - under 1% for documents with over 20% of AI writing."

and describes its validation: "before every … new model release, we perform tests on over 700,000 additional academic papers that were written before the release of ChatGPT to further validate our less than 1% false positive rate."

Now the vendor's own fine print, which is extensive and deserves credit:

• The sentence-level rate is much higher. From the May 2023 update by Chief Product Officer Annie Chechitelli: "Our sentence-level false positive rate is ~4%."
• Low scores are unreliable by Turnitin's own account: "in cases where we detect less than 20% of AI writing in a document, there is a higher incidence of false positives. This is inconsistent behavior, and we will continue to test to understand the root cause." The product now shows "an asterisk (*%) and no percentage" below that 20% threshold.
• The score is not a verdict, per the vendor: "we must emphasize that the percentage on the AI writing indicator should not be used as the sole basis for action or a definitive grading measure by instructors." And from the March 2023 post: "given that our false positive rate is not zero, you as the instructor will need to apply your professional judgment."
• One striking published detail: "54% of false positive sentences are located right next to actual AI writing" — meaning in mixed documents, the detector's errors cluster against the student's own sentences adjacent to flagged ones.

The independent record for Turnitin: in Weber-Wulff et al.'s fourteen-tool study, "Turnitin received the highest score using all approaches to accuracy classification, followed by Compilatio and GPT-2 Output Detector" — the best of the tested field, in a study whose overall conclusion was still that "all scored below 80% of accuracy" and that the tools "should not be used in academic settings." The Washington Post's 2023 hands-on test (16 sample documents; cited here as excerpted by Penn State's Digital Shred toolkit, since the original is paywalled) reported: "It got over half of them at least partly wrong," "accurately identified six of the 16," and flagged "8 percent" of one student's verifiably original essay. In Perkins et al.'s 2024 trial, Turnitin's mean accuracy across conditions was 61%.

Side by side: under 1% document false positives (vendor, specific corpus and threshold) — versus over-half-partly-wrong on a journalist's 16 mixed samples, 61% mean accuracy in an adversarial academic trial, and a ~4% sentence-level error rate from the vendor itself. All of these can be simultaneously true, because they measure different things — which is precisely why the single marketing-page number tells an institution so little. The vendor's own sole-basis warning is the operative sentence.

GPTZero: under 1%, with a conversation-starter posture

GPTZero's technology page claims:

"GPTZero is much better than our competitors at detecting mixed documents where both AI and human writing is involved, with a 96.5% accuracy rate."

"Our false positive rate is under 1%, making it one of the most reliable detectors available."

It also addresses the non-native-speaker problem directly, claiming work that "reduced AI detection's false positive rate on TOEFL texts to 1.1%" — a tacit acknowledgment of the bias findings in Liang et al. And its stated use posture is modest: "We encourage using it as a conversation starter, and not as the final verdict."

The independent record: in Weber-Wulff et al. (testing the 2023-era tool), GPT Zero's false positives were severe enough that "half of the positive classifications would be false accusations, which makes this tool unsuitable for the academic environment." In Perkins et al. 2024, GPTZero's mean accuracy across conditions was 26.3% — the lowest of the named tools in that trial. The RAID benchmark offers the countervailing datapoint: "GPTZero was unusually robust to adversarial attacks."

Side by side: under 1% false positives and 96.5% mixed-document accuracy (vendor) — versus half-of-positives-false-accusations (2023, independent) and 26.3% mean accuracy (2024, independent), alongside genuine adversarial robustness (2024, benchmark). The spread across independent measurements is itself the finding: performance depends enormously on corpus and condition, which no single vendor number conveys.

Copyleaks: the largest gap in this review

Copyleaks' product page claims:

"AI Detector 99% accuracy backed by independent third-party studies"

"With over 99% accuracy and an industry-low .03% false positive rate, Copyleaks leads the field in trusted detection."

with per-language figures such as "English: 99.97% (human), 99.20% (AI)."

The independent record: Copyleaks was not among Weber-Wulff et al.'s fourteen tools (confirmed against the full text), so the relevant peer-reviewed test is Perkins et al. 2024, which reported:

"Notably, despite detecting the highest proportion of manipulated text, Copyleaks possessed the highest likelihood of producing false results, with 50% of human-generated samples misidentified as AI written."

Its mean accuracy across that trial's conditions was 64.8% — the best baseline of the tools tested, bought at the cost of that false-positive rate.

Side by side: a claimed .03% false-positive rate — versus 50% of human control samples misidentified in an independent peer-reviewed trial. That is a gap of more than three orders of magnitude. Methodological differences surely account for part of it (corpus, threshold, document length), but no choice of methodology reconciles .03% with 50%. One of those numbers cannot describe the documents a teacher actually scans.

Originality.ai and the self-evaluation problem

Originality.ai's accuracy page (dated 25 Oct 2026) claims "Originality.ai 5.0 and 5.0 Max have 99%+ accuracy and set the new standard for AI detection," with prior models at "99%+ accuracy in detecting AI content (under 3% false positive rate)." These figures are from the vendor's own testing, on the vendor's own datasets — the exact evaluation pattern RAID was built to correct. RAID's independent note on this tool is partially favorable ("Originality achieved high precision in some constrained scenarios") within a benchmark whose overall verdict was that detectors are "not yet robust enough for widespread deployment or high-stakes use."

The honesty baseline: OpenAI's published failure

One vendor published numbers that made its own product look bad, and they are the most credible numbers in this genre. OpenAI's classifier announcement (January 2023) stated:

"our classifier correctly identifies 26% of AI-written text (true positives) as 'likely AI-written,' while incorrectly labeling human-written text as AI-written 9% of the time (false positives)."

Six months later the same page carried: "As of July 20, 2023, the AI classifier is no longer available due to its low rate of accuracy." The company with the deepest knowledge of its own models' output measured 26% true positives against 9% false positives, and withdrew. Every marketing figure above should be read against this baseline: when the evaluation was run by a party with no incentive to flatter the product, this is what the numbers looked like.

Claims vs measurements, in one table

Tool	Vendor claim (verbatim, their page)	Independent measurement
Turnitin	FPR "under 1% for documents with over 20% of AI writing"; sentence-level "~4%" (own blog)	Best tool in Weber-Wulff (field still "below 80%"); "over half… at least partly wrong" on WaPo's 16 samples; 61% mean accuracy (Perkins)
GPTZero	"false positive rate is under 1%"; "96.5% accuracy" on mixed documents	"Half of the positive classifications would be false accusations" (Weber-Wulff, 2023 tool); 26.3% mean accuracy (Perkins); "unusually robust to adversarial attacks" (RAID)
Copyleaks	"over 99% accuracy and an industry-low .03% false positive rate"	"50% of human-generated samples misidentified as AI written"; 64.8% mean accuracy (Perkins)
Originality.ai	"99%+ accuracy" (own testing)	"High precision in some constrained scenarios" within a failing field (RAID)
OpenAI (retired)	26% true positives, 9% false positives (own published eval)	Withdrawn by the vendor "due to its low rate of accuracy"

What the record does and does not establish

Established:

• Every independent, named-tool measurement in the peer-reviewed literature and the shared benchmark came in far below the figures on the corresponding vendor pages, for every vendor tested.
• The gaps are largest exactly where the stakes are highest: false-positive rates (.03% claimed vs 50% measured for Copyleaks; under 1% claimed vs half-of-positives-false for 2023 GPTZero).
• The vendors' own documentation, read carefully, already concedes the operative point: Turnitin instructs that its score "should not be used as the sole basis for action"; GPTZero positions its product as "a conversation starter, and not as the final verdict."
• The one self-evaluation with no promotional incentive (OpenAI's) produced numbers consistent with the independent literature, not with the marketing pages.

Not established:

• That any vendor fabricated its numbers. Vendor evaluations use their own corpora and thresholds; the claim-measurement gaps demonstrate non-generalization, not necessarily dishonesty.
• Current-version performance. The independent studies tested the tools of 2023–2024; vendors update models continuously (and advertise doing so). What has not changed is the asymmetry of evidence: updated claims are published instantly, independent re-tests arrive years later, and the burden sits on the accused student in the meantime.
• Relative rankings. Tools swap positions across studies and conditions — Copyleaks is simultaneously the best baseline and the worst false-positive producer in the same trial. No stable "most accurate detector" exists in the independent record.

Why this matters for provenance

An institution acting on a detector score is acting on a vendor's number — and this review shows those numbers do not survive contact with independent measurement, by margins that reach three orders of magnitude. The vendors' own pages, to their credit, mostly say the same thing in fine print. Evidence that holds up in a hearing has to be checkable by the people in the room: that means process records, version history, and sealed, independently verifiable writing records — not a percentage whose error rate depends on who you ask.

Related reading: how accurate are AI text detectors? · what your Turnitin AI score actually means · what the research says for integrity panels