What the research says — a briefing for academic integrity panels

Why this briefing exists

A hearing panel asked to weigh a detector score is being asked to weigh an expert instrument — and the instrument has a published error record. This page assembles that record from the peer-reviewed and benchmark literature, with verbatim quotations and links, so a panel can calibrate the weight of the evidence in front of it. Longer close readings of each study live in our research section; this is the version for the hour before a hearing.

Nothing here argues that AI misuse doesn't happen or shouldn't be pursued. It argues one narrower thing, which the literature states outright: a detector score, by itself, is not proof.

Finding 1: measured accuracy is below what an accusation requires

The largest independent academic evaluation (Weber-Wulff et al., 2023; fourteen tools including Turnitin and GPTZero) concluded:

"Detection tools for AI-generated text do fail, they are neither accurate nor reliable (all scored below 80% of accuracy and only 5 over 70%)."

And in its conclusion:

"Therefore, our conclusion is that the systems we tested should not be used in academic settings."

A 2024 trial in a higher-education setting (Perkins et al., six detectors, 805 samples) measured 39.5% baseline accuracy, falling to 17.4% on manipulated text, and concluded the tools "cannot currently be recommended for determining whether violations of academic integrity have occurred."

Weight for the panel: the independent literature, not the vendor documentation, is the record a decision will be measured against on appeal.

Finding 2: the errors are not evenly distributed

The Stanford study (Liang et al., Patterns 2023) ran seven widely used detectors on human-written TOEFL essays and US eighth-grade essays:

"While the detectors accurately classified the US student essays, they incorrectly labeled more than half of the TOEFL essays as 'AI-generated' (average false-positive rate: 61.3%)."

19.8% of those human essays were flagged by all seven detectors unanimously — so "multiple tools agree" is not the safeguard it appears to be. Weber-Wulff et al. found the related effect for translation: human text machine-translated into English lost 20 points of accuracy.

Weight for the panel: if the accused student writes in English as a second language, the prior probability of a false positive is materially higher, and tool-consensus does not repair it. Details: non-native speakers and detector bias.

Finding 3: evasion is cheap, so a clean score clears no one and a flagged score convicts no one

Weber-Wulff et al.: unmodified AI text was caught at 74%; lightly human-edited AI text at 42%; machine-paraphrased AI text at 26% — "most AI-generated texts remain undetected when machine-paraphrased." The attack literature reproduces this with stronger tools (a research paraphraser dropped one detector from 70.3% to 4.6% detection at a 1% false-positive rate — see the paraphrase-attack review).

The ACL 2024 shared benchmark (RAID, 6M+ generations, 12 detectors) states the combined verdict:

"Detectors are not yet robust enough for widespread deployment or high-stakes use: many detectors we tested are nearly inoperable at low false positive rates, fail to generalize to alternative decoding strategies or repetition penalties, show clear bias towards certain models and domains, and quickly degrade with simple black-box adversarial attacks."

Weight for the panel: detector evidence is weak in both directions. It cannot clear a sophisticated cheater, and it cannot convict an unlucky honest writer. Its evidentiary value is a prompt for inquiry, not a result of one.

Finding 4: the model vendors themselves say so

Two facts from the companies that build the generators:

• OpenAI retired its own AI-text classifier in July 2023 "due to its low rate of accuracy" (its stated reason), and explains that it has withheld its text watermarking system partly because it "could stigmatize use of AI as a useful writing tool for non-native English speakers" and because "applying it to large volumes of text would lead to a large number of total false positives."
• Google shipped a text watermark (SynthID, in Gemini) and states in the accompanying Nature paper: "No text detection method is foolproof." Its own blog adds that confidence "can be greatly reduced when an AI-generated text is thoroughly rewritten or translated to another language." (Full review: watermarking.)

Weight for the panel: when the parties with the most capability and the most incentive publish these caveats about their own technology, a third-party score deserves at least that much skepticism.

Questions a panel should put to detector evidence

1. What false-positive rate was this score measured at — and who measured it? The RAID benchmark's central finding is that accuracy claims are uninterpretable without this number, and default settings produce false-positive rates it calls "dangerously high."
2. Is the student a non-native English writer, or was any part of the workflow translated? If yes, Finding 2 applies directly.
3. Would this evidence survive the published error rates being read into the record? Imagine the student's advisor quoting "should not be used in academic settings" from the integrity field's own journal.
4. What does the tool's own documentation disclaim? Most vendors state the score should not be the sole basis for action; a process that treats it as such contradicts the instrument's instructions.
5. Is there process evidence? Drafting history, version records, or a verifiable writing record either corroborates the student or doesn't — and unlike the score, it is evidence about this document, not a statistical guess about documents like it.

What stronger evidence looks like

The pattern across the literature is that output-based inference fails because the output is cheap to reshape. Process-based evidence does not share that weakness: a drafting history shows when text arrived, in what sizes, with what revision structure. For how to evaluate such records — including their real limits — see how to read a process record, process data in integrity cases, and, on the student-facing side, Google Docs version history as proof. For redesigning assessment so the question arises less often: alternatives to AI detectors and designing writing assignments when detectors fail.

A score says: documents with these statistics are often AI-generated. A process record says: this document was written like this. Panels decide individual cases, and only one of those statements is about the individual case.

Related reading: how accurate are AI text detectors? · detector benchmarks — RAID and M4 · how to talk to a student about suspected AI use