How AI detectors actually work

Perplexity, burstiness, and why every detector reports a probability rather than an answer.

Detection · 2 min read

AI detectors do not find a hidden mark. With one exception — Google's SynthID, where the key holder can check a real embedded signal — they analyse the text itself and guess.

Understanding the guess explains both why they sometimes work and why they fail in the specific ways they do.

Perplexity

Perplexity measures how surprising a text is to a language model. Run the text through a model and ask, at each word, how probable that word was given everything before it. Consistently high-probability words mean low perplexity.

Model output tends to have low perplexity, because sampling favours likely continuations. Human writing tends to have higher perplexity — people choose odd words, make unexpected turns, occasionally err.

So the crude heuristic is: unusually unsurprising text is probably machine-written.

Burstiness

Burstiness measures variation in that surprise across a document. Human writing is bursty — a dense clause, then a plain one, then a fragment. Model output is flatter, holding a fairly even level throughout.

Low perplexity plus low burstiness is the classic signature detectors look for.

Why this fails in both directions

False positives hit predictable groups hardest:

• Non-native English writers, who use more common vocabulary and more regular constructions. Their writing has genuinely lower perplexity. This is the most documented failure mode and it is a fairness problem, not a bug to be tuned away.
• Technical and academic writing, which is deliberately formulaic. Method sections, legal boilerplate, and standardised reports all read as low perplexity because the register demands it.
• Heavily edited writing. Copy editing removes exactly the surprising choices that raise perplexity. Well-edited human prose looks more machine-like than a rough draft.

False negatives are trivially achieved. Light editing raises perplexity. So does asking the model to write less formally. Detection is easiest against unedited default-voice output and degrades fast from there.

Why the scores disagree

Different detectors use different reference models, thresholds, and training data. The same passage routinely gets 90% from one tool and 15% from another. Neither is measuring a ground truth — each is reporting how unusual the text looks to its particular model.

This is why any product claiming to guarantee it will "pass" a named detector is making a claim it cannot support. The detector can change its model tomorrow.

What is actually knowable

Two things, and it is worth being precise about the difference:

Verifiable: the exact count of hidden characters, exotic spacing and typographic punctuation in a document. This is character matching. The answer is a number, not a probability.

Not verifiable: whether any given detector will flag your text. That depends on proprietary systems that change without notice and disagree with each other.

We will tell you the first exactly and make no promises about the second. Run a scan and see.

Related: why detectors produce false positives · what SynthID does