Put a Confidence Score on Every Extraction

Most extraction code has exactly two outcomes: it parsed, or it threw. That binary hides the most dangerous class of records — the ones that parsed cleanly and are still wrong. A number that landed in the right field but off by an order of magnitude. A date that coerced successfully into the wrong year. Structurally valid, semantically garbage, and completely invisible to a try/except.

The fix is to stop treating extraction as pass/fail and start treating it as a measurement with an uncertainty attached. Every extracted record should carry a confidence score.

What the score buys you

A confidence signal changes the operational question from "do I trust the model?" to "which records do I trust?" That's the difference between two unscalable extremes:

• Trust everything. Fast and cheap until the first silent-corruption incident, which you find weeks later in a downstream report.
• Review everything. Safe and completely non-scalable; you've reinvented manual data entry with extra steps.

With a per-record score you get the third option: triage. Set a threshold. High-confidence records flow straight through. Low-confidence ones route to human review, a second pass, or a dead-letter queue. Now human attention scales with risk, not volume — which is the only version of this that survives real traffic.

This is a first-class part of how confident-extract works: v0.1.0 attaches a confidence score to every result, so routing-by-confidence is a property of the extraction step rather than something you bolt on afterward.

Thresholds are a product decision, not a constant

The temptation is to hard-code if confidence > 0.9. Resist it. The right threshold depends on the cost of being wrong for that field in that workflow:

• A field that feeds an irreversible action (a payment, an auto-approval) deserves a high bar and cheap escalation.
• A field that's advisory, or trivially correctable downstream, can run looser.
• Review capacity is finite, so the threshold is also a throughput knob: it sets how much lands on a human's desk.

Treat the threshold as configuration, monitor the review-queue rate, and tune it like any other production dial.

Calibration: the honest caveat

A confidence score is only useful if it means something — if "0.9" records really are right about nine times in ten. That property is calibration, and it's worth being clear-eyed about: a score is a routing signal, not a mathematical guarantee. The way you keep it honest is by closing the loop. Sample the records that human review corrected, compare them against the scores they carried, and watch whether the ranking holds. If low scores reliably catch the errors, the signal is doing its job even if the absolute numbers drift.

That feedback loop is also your early warning system. When the confidence distribution shifts — more records piling up near the threshold, a field that used to score high now scoring low — something upstream changed: a new document template, a model version, a schema that drifted. (I wrote about that failure mode in Schema Drift Is the Silent Killer of LLM Pipelines.)

The mindset shift

Adding a confidence score is a small API change with a large operational payoff. It moves a pipeline from hoping the model was right to knowing which outputs to double-check — and it makes "human in the loop" a scalable, targeted mechanism instead of a bottleneck. In production AI, that's most of the game: not eliminating uncertainty, but measuring it and routing around it.