Can writing style identify an author? Stylometry from the Federalist Papers to LLMs

What this review covers

Long before anyone asked "did an AI write this?", researchers asked "did this person write this?" — and built a discipline around answering it from style alone. Stylometry quantifies the habits a writer cannot easily suppress: function-word rates, sentence-length rhythms, punctuation choices, character n-grams. It is the intellectual ancestor of every AI detector, and its sixty-year track record is the best available evidence for what style-based identification can and cannot do.

This review reads five documents: the field's most rigorous recent benchmark-of-benchmarks (Tyo, Dhingra & Lipton, 2022), the major survey of authorship attribution in the LLM era (Huang, Chen & Shu, 2024, revised 2026), the paper that first showed where stylometry breaks against machine text (Schuster et al., 2019), an EMNLP 2024 study of whether LLMs can themselves do attribution, and — for the historical note only — the tertiary record of the field's founding result. Every quotation is verbatim from the linked source, checked on the asOf date.

Why it belongs in a provenance research silo: "compare the submitted essay against the student's earlier writing" is a stylometric claim, and institutions reaching for it deserve to know what the literature says about its accuracy.

Where the field comes from

The origin story is real and checkable. Tyo et al.'s survey dates it precisely:

"Computer-assisted stylometry first emerged in the early 1960s, when Mosteller and Wallace 1963 explored the foundations of computer-assisted authorship analysis."

The problem Mosteller and Wallace solved is the field's founding legend. Per the tertiary record (Wikipedia, cited here only for this historical note): "One notable early success was the resolution of disputed authorship of twelve of The Federalist Papers by Frederick Mosteller and David Wallace." Twelve essays, claimed by both Hamilton and Madison for 150 years, attributed by statistics on the writers' unconscious word-rate habits — a result historians broadly accepted.

Huang, Chen & Shu's survey traces the arc since:

"Stylometry, the quantitative analysis of writing style, has evolved from its initial reliance on human expertise (Mosteller and Wallace, 1963) to computational methods..."

Note the shape of the founding problem, because it defines where stylometry is strong: a closed set (exactly two candidate authors), large samples of undisputed text from each, and a shared genre (political essays from the same project). Most of the field's failures since are departures from one of those three conditions.

Tyo et al. (2022): the honest state of the art

The most useful accuracy numbers come from Tyo, Dhingra & Lipton's "On the State of the Art in Authorship Attribution and Authorship Verification" — a re-evaluation that ran the leading method families on a standardized suite of tasks. (Attribution: which of these candidate authors wrote this? Verification: did this specific author write this?)

The headline, verbatim:

"Surprisingly, we find that a traditional Ngram-based model performs best on 5 (of 7) AA tasks, achieving an average macro-accuracy of 76.50% (compared to 66.71% for a BERT-based model). However, on the two AA datasets with the greatest number of words per author, as well as on the AV datasets, BERT-based models perform best."

Three readings:

• The ceiling is around 76.5%, averaged across attribution tasks, for the best method class — after sixty years of work. That is far from random, and far from courtroom certainty.
• Old methods beat new ones when data is thin. Character n-grams won 5 of 7 tasks; deep models only pulled ahead "on the two AA datasets with the greatest number of words per author." Style identification is data-hungry: the deciding variable is how much text per author you have — which, for a student with a handful of prior essays, is very little.
• The field's own evaluation hygiene is a problem. The authors note that "inconsistent dataset splits/filtering and mismatched evaluation methods make it difficult to assess the state of the art" — an echo of the shared-benchmark problem in AI detection (see RAID and M4).

Schuster et al. (2019): where stylometry breaks against machines

Before ChatGPT, Schuster et al. asked whether stylometry could catch machine-generated misinformation. Their answer defined the boundary that AI detection has been living with since. Verbatim from the abstract:

"we show that stylometry is limited against machine-generated misinformation. While humans speak differently when trying to deceive, LMs generate stylistically consistent text, regardless of underlying motive. Thus, though stylometry can successfully prevent impersonation by identifying text provenance, it fails to distinguish legitimate LM applications from those that introduce false information."

Unpack the mechanism, because it is the deepest sentence in this review. Stylometry works on humans because style leaks involuntarily — a deceiving human writes measurably differently from an honest one. A language model has no such leak: its style is uniform whatever its "motive." Style can tell you which pen wrote the text; it cannot tell you why, and against machine text it increasingly cannot even tell you whose pen, because the machine writes in everyone's style and no one's.

Huang, Chen & Shu (2024): attribution in the LLM era

The current survey of record states the field's new condition plainly:

"LLMs have blurred the lines between human and machine authorship, posing significant challenges for traditional methods."

and:

"LLMs can emulate human writing so closely that they pose significant challenges to traditional stylometric techniques."

The survey is also candid about the unsolved engineering problems that predate LLMs. On what a usable attribution system requires:

"Generalization requires the ability to generalize across various domains, while explainability emphasizes providing transparent and understandable insights..."

Cross-domain transfer — recognizing an author's lab report from their personal essays — remains hard; the survey cites prior work that "concluded that topic diversity and dataset size are crucial for effective cross-domain transfer." A student's prior writing rarely offers either. And on the adversarial dynamic:

"As detection methods advance, adversarial attacks also evolve to bypass these measures, creating a continuous cycle of challenge and response in the quest to distinguish and disguise authorship..."

A companion study by the same group — "Can Large Language Models Identify Authorship?" (EMNLP 2024 Findings) — tested whether LLMs themselves can do zero-shot attribution, motivated by the stakes: "The ability to accurately identify authorship is crucial for verifying content authenticity and mitigating misinformation." The experiments test verification and attribution among pools of ten to twenty candidate authors — closed sets, like every strong stylometry result since 1963.

The literature, side by side

Source	Question	Finding
Mosteller & Wallace 1963 (via Tyo et al.; Wikipedia for the historical note)	Hamilton or Madison? (2 candidates, large corpora)	Field-founding success; the template for when stylometry works
Tyo et al. 2022	best current accuracy across standardized tasks	76.50% macro-accuracy (n-grams) vs 66.71% (BERT); deep models win only with lots of text per author
Schuster et al. 2019	can style catch machine-generated deception?	"stylometry is limited against machine-generated misinformation"; style identifies the pen, not the motive
Huang, Chen & Shu 2024 (survey)	state of the field under LLMs	"LLMs have blurred the lines between human and machine authorship"; cross-domain transfer and explainability unsolved
Huang, Chen & Shu 2024 (EMNLP Findings)	can LLMs do attribution themselves?	tested on closed sets of 10–20 candidates — the classical regime, now with new models

What the literature does and does not establish

Established:

• Stylometry genuinely works in its home regime: closed candidate sets, substantial undisputed text per author, matched genre. The Federalist result and six decades of replication are real.
• The honest state-of-the-art number, measured across standardized attribution tasks, is roughly 76.5% — strong evidence, nowhere near certainty, and achieved by simple n-gram models except where per-author data is unusually plentiful (Tyo et al.).
• Accuracy depends on the amount of prior text per author more than on model sophistication (Tyo et al.).
• Against machine-generated text, the field's own papers say the signal degrades: models "generate stylistically consistent text, regardless of underlying motive" (Schuster et al.), and LLMs "can emulate human writing so closely that they pose significant challenges to traditional stylometric techniques" (Huang, Chen & Shu).
• Cross-domain attribution — matching an author across genres — remains an open problem dependent on "topic diversity and dataset size" that typical real-world samples lack.

Not established:

• That style comparison can reliably decide an individual integrity case. No paper in this set evaluates the realistic classroom setting: one disputed essay, a few prior samples in different genres, an open set (the "other author" might be any LLM, any human, or a mixture), and a writer whose style legitimately changes between drafts, topics, and years.
• Any specific error rate for "this essay doesn't match the student's previous work" claims. The 76.5% figure comes from curated closed-set benchmarks; the literature reviewed here contains no validated number for the forensic-education scenario.
• That LLM-assisted attribution escapes the classical constraints — the current experiments still operate on closed sets of 10–20 candidates.

Why this matters for provenance

Stylometry is what "we compared it to your previous essays" actually is, and its own literature prices that evidence honestly: strong in a two-candidate closed world with lots of text, ~76% in standardized benchmarks, and explicitly challenged by the existence of models that write "in everyone's style and no one's." Like the detectors it fathered, it infers the past from the artifact. A process record does not infer: it was there. See what writing provenance is, how to read a process record, and keystroke logging and writing-process research — the branch of this literature that measures the writing event rather than its residue.

Related reading: how accurate are AI text detectors? · detector benchmarks — RAID and M4 · paraphrase attacks on AI detectors