In MightyBot’s production lending deployment, 99%+ accuracy is a system property, not a model property. No prompt, frontier model, or voting ensemble produces it alone because the remaining errors are architectural: variance, unverified data, and unmanaged ambiguity. This post dissects five subsystems, each closing a class of error the others cannot.
Why accuracy plateaus
Teams discover the plateau when a strong demo degrades in production and every fix buys less than the last one. The reason is that “accuracy” aggregates three different failure classes:
- Variance errors. The same case, run twice, decides differently. Reasoning loops improvise their path per run; borderline cases flip. A measured study put the single-pass borderline flip rate near 20%.
- Data errors. The decision logic was right; the number it evaluated was wrong: misread, misattributed, or fabricated during extraction.
- Judgment errors. Genuinely ambiguous cases where the model’s call differs from what your senior reviewer would decide.
Voting attacks only the first class and leaves a persistent borderline error floor. Better models help the third class and leave the first two intact. Reaching 99% means closing all three, which takes structure.
The five subsystems
1. Deterministic execution. Compile the workflow into a fixed plan: same inputs, same path, every time. Every rule that can run as code runs as code, including threshold checks, reconciliation, and completeness, contributing zero variance and zero tokens. Model calls happen where judgment is required, schema-bound and scoped. This is the Agent Compiler’s output, and it eliminates the variance class at the source.
2. Evidence-linked extraction. Every value the workflow evaluates carries a pointer to its origin: document, page, character offset. Extraction without provenance is unverifiable by construction; with it, wrong values become catchable, by validation code or by a reviewer in seconds. The document pipeline classifies, extracts, normalizes, and reconciles across sources before policy ever runs, which is where the data-error class dies.
3. Confidence routing. The system knows what it could not verify. Low-confidence classifications, failed reconciliations, and rule-flagged conditions route to a human with the evidence attached; everything verified proceeds. This converts judgment errors from silent to managed: they arrive at a reviewer as exceptions instead of leaving as mistakes.
4. Review gates and progressive autonomy. Accuracy is earned in stages: agents run in audit mode while humans keep deciding, graduate to assist, then to automation on clean cases, with autonomy raised on evidence about how often reviewers change agent output. The gate placement is policy, owned by compliance, not engineering.
5. Closed-loop correction. Every human correction is data. When a reviewer fixes an edge case, the policy is refined in plain English, backtested against history, and deployed; the same mistake does not recur quarterly. Accuracy in month twelve is higher than month one because the loop compounds.
The honest arithmetic
MightyBot’s production lending deployment sustains 99%+ correct decisions, with roughly 2% of cases routed to human review across more than $100B in lending activity. What is left for humans is the 2% that deserves attention, not 100% reviewed out of fear.
That is also the economic story. Every point of accuracy below the human baseline converts to residual review labor; the difference between an 84% agent and a 99% system, priced at production volume, usually exceeds the entire platform cost. The ROI calculator models that trade, with the accuracy trajectories of each build path charted against time.
Accuracy is not a model you pick. It is a system you either build over time or buy running.
