Superuser.
Insights · AI accuracy

AI made unfinished thinking look finished.

Modern AI is fluent, fast, and confidently wrong just often enough to be dangerous. In high-stakes work, the problem isn't that AI gets things wrong — it's that the wrong answer looks exactly like the right one. Here's what's actually going on, and how to ship safely.

Confident
Wrong answers sound right
0 sources
Most claims aren't cited
Pre-send
Cheapest place to catch it
The core problem

What a hallucination really is

A "hallucination" is when an AI produces a claim that is fluent, plausible, and completely unsupported by any source. It isn't lying — it has no concept of true or false. A language model predicts the most likely next words, so it fills gaps with text that sounds correct rather than text that is correct.

That's why hallucinations are so hard to spot: there's no grammar mistake, no obvious tell. A fabricated statistic, an invented case citation, or a number that drifted by a decimal place reads with the same calm authority as a fact.

Where accuracy breaks

The five places it bites in real work

  • Numbers. Figures get rounded, transposed, or quietly invented — and a bracket that means 'negative' gets read as positive.
  • Citations. References to cases, papers, or sources that don't exist, or that say the opposite of the claim.
  • Assumptions. Confident conclusions built on a premise the model assumed but never stated.
  • Claims. Statements presented as established fact with nothing behind them.
  • Caveats. The risk, the exception, the 'subject to' line that a human would always add — silently dropped.
Why it's costly

Fluency hides the gaps

Before AI, a rough draft looked rough — so it got a proper review. AI removes that signal. Unfinished thinking now arrives polished, which means the reviewer relaxes exactly when they should lean in. The error doesn't get caught at the desk; it gets caught by the client, the board, the regulator, or the market — where it's far more expensive.

How to ship safely

Trust the words, prove the facts

The fix isn't to stop using AI — it's to verify before you send. The principle we build on: code proves the numbers, AI reviews the words. Anything that can be checked deterministically (a calculation, a total, a sign) should be proven, not estimated. Anything softer (a claim, a missing caveat) should be flagged for a human, never silently rewritten.

Provable

Numbers and math, checked deterministically.

Flaggable

Unsupported claims and missing caveats, surfaced.

Judgment

The final call stays with a human.

See it catch a real gap

Look at sample pre-send checks — the numbers, claims and caveats that nearly went out.

This article is general guidance, not legal, financial, or professional advice. Superuser supports human review — it does not replace it.