Ten steps at 95 percent is a 60 percent agent

Per-step accuracy multiplies, so a long agent run fails more often than any single step suggests. The fixes are structural: fewer model steps, checks, saved state, approval.

Task Planner Executor Verifier Checkpoints Approval gate Irreversible action Run log check in code a person signs off every step saved state retry from last checkpoint
A failed check sends the run back to the executor from the last checkpoint, and a person approves before the irreversible action.
On this page
  1. The multiplication, written out
  2. Prompt edits run out before 99
  3. Four changes that aren’t in the prompt
  4. An invoice agent stuck at 58 percent
  5. Where the retry stops helping

Multiply 0.95 by itself ten times and you get 0.599. An agent that gets each step right 95 times in 100 finishes a ten-step task about 60 times in 100, and 95 percent per step is a number most teams would be pleased with.

I think this explains what happened to the autonomous agents of spring 2023 better than any complaint about the models. Auto-GPT and the projects around it let a model plan its own steps and run them in a ReAct-style loop until it declared itself done. Any single step looked reasonable. Thirty in a row rarely landed.

The multiplication, written out

Assume steps fail independently. That’s generous, since a wrong step also hands bad input to the next one. End-to-end success is then per-step accuracy raised to the number of steps.

Per-step accuracy 5 steps 10 steps 20 steps
90% 59% 35% 12%
95% 77% 60% 36%
98% 90% 82% 67%
99% 95% 90% 82%

Read along the 95 row and doubling the steps takes you from 60 to 36. Read down the ten-step column and the jump you need, from 60 to 90, means going from 95 to 99 per step. That’s a five-fold cut in the error rate, on every step at once.

Prompt edits run out before 99

The first rewrite of a weak prompt fixes the common failure. What’s left afterward is the odd input: the scanned invoice with a rotated table, the vendor whose name shows up twice with different spellings. Each needs its own fix, and a new model version reshuffles the set.

So I’d stop treating a 60 percent run as a wording problem. There are two numbers in that formula, and the exponent is the one an engineer controls.

Four changes that aren’t in the prompt

Start by taking steps away from the model. Anything code can do exactly, code should do. Function calling made it easy to hand the model a fetch tool and then let it decide when to fetch, which turns a call that never fails into one that fails 3 percent of the time. Every step moved to code drops out of the exponent.

Then check every step the model still owns, with code. I mean a deterministic check. A second model’s opinion doesn’t count. Do the line totals sum to the invoice total? Does the purchase order number exist? When a check fails, retry once and include the failure message. It’s the idea I described for tool arguments, applied to a whole step’s output.

Save state after each verified step, so a failure at step eight resumes at step eight. Rerunning one to seven means rolling the dice on them again. LangGraph’s checkpointers do this, and the library reached a stable 0.1 in June. A table keyed by run ID and step number does it too, if you own the loop.

The last change is a person in front of anything that can’t be undone. LangGraph calls this an interrupt: the graph pauses before a named node, the state waits in the checkpoint store, and the run continues when someone approves. The gate depends on the checkpoint. Without saved state, an approval that arrives three hours later has nothing to resume.

Laid end to end, a run goes from planner to executor to verifier, into the checkpoint store, and through an approval gate before the irreversible action. A run log sits at the far end.

An invoice agent stuck at 58 percent

Wexcombe Procurement is fictional. I picked its success rates so the multiplication stays readable.

Wexcombe runs a ten-step agent that matches supplier invoices to purchase orders and posts them to the ledger. The model decides every step. Per-step success sits between 94 and 99 percent, and over 300 runs the agent finishes correctly 174 times. That’s 58 percent.

The team fills in a step table and finds four steps that never needed a model: fetching the invoice, looking up the purchase order, the tolerance arithmetic, and the posting call. Those go to code. The six that remain each get a verifier and one retry.

# One entry per step. Fill it in before touching a prompt.
run: invoice-match
steps:
  - step: fetch_invoice
    decided_by: code
    verifier: file opens and has at least one page
    reversible: true
    checkpoint: true
    approval: none
  - step: extract_header
    decided_by: model
    verifier: invoice number, date and total present and well formed
    reversible: true
    checkpoint: true
    approval: none
  - step: extract_lines
    decided_by: model
    verifier: line totals sum to the invoice total
    reversible: true
    checkpoint: true
    approval: none
  - step: resolve_vendor
    decided_by: model
    verifier: vendor ID exists and is active
    reversible: true
    checkpoint: true
    approval: none
  - step: lookup_purchase_order
    decided_by: code
    verifier: purchase order exists for that vendor
    reversible: true
    checkpoint: true
    approval: none
  - step: match_lines
    decided_by: model
    verifier: every invoice line maps to one open order line
    reversible: true
    checkpoint: true
    approval: none
  - step: check_tolerance
    decided_by: code
    verifier: none needed, plain arithmetic with unit tests
    reversible: true
    checkpoint: true
    approval: none
  - step: classify_exception
    decided_by: model
    verifier: reason code is in the allowed list
    reversible: true
    checkpoint: true
    approval: none
  - step: draft_posting
    decided_by: model
    verifier: debits equal credits and accounts exist
    reversible: true
    checkpoint: true
    approval: none
  - step: post_to_ledger
    decided_by: code
    verifier: ledger returns a posting ID
    reversible: false
    checkpoint: true
    approval: required above 10000 USD

With verifiers and a retry, the six model steps come out at about 98.5 percent each. Six steps at 98.5 is 91 percent, so 273 good runs in 300 where there were 174. Postings above $10,000 wait for a person.

Where the retry stops helping

A verifier only catches what it can check. If a 94 percent step had a perfect verifier, one retry would lift it past 99.6. Wexcombe gets 98.5 because some wrong answers pass: line items that sum correctly but are matched to the wrong order line get through the total check.

That’s what the run log is for. Every verifier failure and every retry goes into it with the step name, and after a few hundred runs it shows which step is eating the retries. That step gets a stricter check or a move to code.

In Wexcombe’s final table, no step pairs decided_by: model with reversible: false. I wouldn’t ship an agent whose table has that pairing anywhere.