Methodology

What The Detector Measures

DraftFilter computes perplexity-ratio style signals (the Binoculars approach): it compares how expected the submitted text looks to a base language model against how expected it looks to an instruction-tuned model from the same family. Text that looks unusually easy for the instruction-tuned model relative to the base model is treated as more AI-like. A logistic ensemble combines these signals from the same forward passes, and Platt scaling maps the ensemble score onto a calibrated scale.

Before scoring, input text is hardened: Unicode normalization plus homoglyph and zero-width character stripping, which neutralizes common character-swap evasion. A memorized-text guard checks submissions against canonical public-domain corpora; near-duplicates of famous texts are never flagged as AI.

The internal detector score runs in the opposite direction from the public API: lower internal scores are more AI-like. Public `is_ai` values convert that internal score into a percentage where higher means more AI-like.

How Thresholds Are Set, And Why We Never Report Raw Accuracy

A single “accuracy” number on a balanced test set hides the error that matters most: how often human writing gets flagged as AI. Independent evaluations have repeatedly shown that headline accuracy claims in this market hold only at high false-positive rates. So we report the true-positive rate (TPR: the share of AI text correctly flagged) at fixed false-positive rates (FPR: the share of human text wrongly flagged) of 1% and 5%, per domain, and we publish the realized FPR on the held-out test split next to every number.

Decision thresholds are fit on held-out human text only (the RAID benchmark protocol), separately per document-length bucket (boundaries at 128, 256, 384 scored tokens), because false-positive behavior shifts with length. Thresholds are never fit on the test data reported below.

Measured Performance

Numbers below come from our committed evaluation report (data/detector_eval/recalibration-report.yml, July 2026) using the production scoring rule: pooled Platt calibration with per-length-bucket thresholds. The corpus combines a RAID benchmark subset (GPT-4-era generators, including adversarial attacks), HART subsets (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3.3), a frontier slice we generated with current frontier models, and our small internal set.

Pooled across all test sets (6132 human / 6162 AI documents): TPR 15.4% at the 1% FPR target (realized FPR 0.8%) and TPR 28.5% at the 5% FPR target (realized FPR 5.3%), AUROC 0.80. These are modest numbers and we publish them anyway: most AI text in hard, adversarial benchmarks is not caught at strict false-positive budgets.

The calibration that fixed low-FPR behavior cost pooled recall at the looser target: before it, pooled TPR was 41.4% at 5% FPR but only 3.4% at 1% FPR. The production rule raises TPR at 1% FPR to 15.4% and keeps per-length false-positive rates stable, at the cost of pooled TPR at 5% FPR dropping to 28.5%. We chose the low-false-positive side of that tradeoff.

RAID subset, per domain

Per evaluation set

Realized FPR is measured on the test-split human documents of each slice; small slices (the frontier slice and the internal set) have wide error bars — a single document moves those percentages by points, not decimals.

Known Limitations

• Most AI text is not flagged at strict thresholds. At a 1% FPR budget the detector catches a minority of AI-generated documents (pooled TPR 15.4%). Treat a low `is_ai` score as absence of evidence, not evidence of human authorship.
• Mixed authorship is not separable. Human drafts edited by AI, and AI drafts edited by humans, land between the calibrated extremes; the detector reports a single document-level signal and cannot attribute individual sentences reliably.
• Humanizer and paraphrase tools degrade detection.Text run through paraphrasers or “humanizer” tools is flagged at a lower rate; the RAID subset above includes such adversarial rewrites, which is part of why its numbers are low.
• Long documents are judged on a window.Scoring truncates at the first 512 scored tokens, so a long document's verdict reflects its opening, not its entirety.
• Short texts are noisy. The under-128-token length bucket has the least calibration data; scores on very short submissions are the least reliable.
• Memorized text is never flagged. Near-duplicates of canonical public-domain works match the memorized-text guard and always return not-AI, by design.
• Generators evolve. These numbers are a snapshot against the generators listed above; new model releases change detection behavior and we recalibrate against them over time.

Benchmarks And External Validation

We evaluate against public benchmarks (RAID and HART subsets) plus a frontier-generation slice built by regenerating RAID's human documents with current frontier models. A submission of this detector to the public RAID leaderboard — scored by the benchmark maintainers, not by us — is in preparation; this page will link to the entry once it is live.

Banding

Public responses include `likely_human`, `mixed`, and `likely_ai` bands. The current calibrated cut-points in data/binoculars_gemma/calibration.yml classify public `is_ai` scores at 35 or below as `likely_human`, scores from 36 through 64 as `mixed`, and scores at 65 or above as `likely_ai`.

Sentence heatmaps use the same public direction: higher `is_ai` means the strongest red highlight; human-sounding sentences get no highlight.

Data Retention

DraftFilter retains all text submitted to the service, including web scanner submissions, anonymous scans, signed-in dashboard traffic, and API requests.

We use retained text and related metadata to operate, secure, measure, and improve AI detection and rewrite quality.

This includes text submitted through the scanner, Humanizer, synchronous API calls, queued jobs, uploads, dashboard history, examples, and client integrations.