A reasoning model copies an invoice number out of a PDF exactly as well as a fast model does. It just takes most of a minute longer to do it.
That’s the trade when one runs a whole pipeline. OpenAI’s o1 has been in the API since December, DeepSeek-R1 put open weights for the same idea on the table in January, and the tempting move is to swap one in for the model you had and call the pipeline smarter. Some of it does get smarter. Most of it gets slower and the answers don’t change, because most steps in a pipeline were never hard.
I treat a reasoning model as a stage with a narrow job description. It gets the step where a wrong answer costs real money and where the input can be made small. Everything else stays on a fast model.
Thinking is billed, and so is waiting
A reasoning model spends tokens before it answers. You pay for those as output tokens whether or not you ever see them, and you wait while they’re produced. The amount grows with how much there is to think about, which is partly the difficulty of the question and partly the size of what you handed over. Give it six thousand tokens of scanned invoice and it will dutifully consider all of them.
So the two things I look at for each stage are the cost of an error and the size of the input. Extraction has a large input and a cheap error, since a later check usually catches a misread field. Anything a person watches in real time has a latency budget that rules the slow model out before accuracy comes up.
What’s left is a short list: planning a multi-step job, reconciling evidence that disagrees, sometimes checking a judgment call before it leaves. Those steps share a property. They take a small, already-structured input, and the answer depends on holding several facts against each other.
The 54 disputes where the documents disagree
Sedgemoor Supply is a distributor I made up, along with its disputes and its numbers. Its customers dispute invoices, usually over a quantity or a price. The pipeline reads the invoice, the purchase order, the receiving record and the customer’s note, then proposes a resolution. The test set is 200 disputes that people have already resolved.
Three ways to build it:
| Pipeline | Resolved correctly | Median latency | p95 latency | Relative cost per dispute |
|---|---|---|---|---|
| Fast model on every stage | 81% (162 of 200) | 2 s | 4 s | 1x |
| Reasoning model on every stage | 93% (186 of 200) | 38 s | 71 s | 30x |
| Split, 27% routed to reasoning | 92% (184 of 200) | 3 s | 44 s | 5x |
The fast-only pipeline gets the easy disputes right and falls over where the documents disagree: the PO says 40 units, the receiving record says 36, and the customer’s note mentions a partial return nobody logged. The reasoning-only pipeline fixes most of those and makes every other dispute wait around 38 seconds for an answer the fast model already had.
The split sends 54 of the 200 disputes to the reasoning stage. It gives up two cases against reasoning-only, both of them disputes the gate judged easy and the fast path got wrong. In exchange the median drops to 3 seconds and the bill to a sixth, because the reasoning stage reads 900 tokens of extracted fields where the reasoning-only pipeline read all four documents. The p95 column is the catch. The flagged disputes still take most of a minute, so the split only works if that 27% can tolerate it.
The gate is code
The piece that makes the split work is the difficulty gate, and I wouldn’t use a model for it. Asking a fast model whether a case is hard gets you a confident guess from the same model that fails on hard cases. A confidence threshold works for labelling email, where the model is scoring its own one-word answer. Here the difficulty sits in the documents, and code can read it directly.
The fast extractor returns a strict schema through Structured Outputs: amounts, quantities, dates, and the document each one came from. The gate is plain code over those fields. If the PO, the invoice and the receiving record disagree on a quantity or a price beyond a tolerance, or the customer’s note references a document that isn’t in the set, the dispute is flagged. No disagreement, fast path.
That same schema is the hand-off. The reasoning stage never sees the PDFs. It gets around 900 tokens of extracted fields and the specific lines in conflict, and it returns a decision, an amount and the evidence it relied on, in another strict schema. If the reasoning model you pick can’t constrain its output to a schema, let it answer in prose and have the fast model format the conclusion.
The verifier runs on both paths and is mostly arithmetic: the proposed credit has to equal the difference the evidence supports, and every cited line has to exist. Failures go to a person.
One entry per stage, before any model is picked
This is the worksheet for the dispute pipeline.
# Stage-placement worksheet: one entry per stage, filled in before choosing models.
pipeline: invoice-disputes
test_set: 200 resolved disputes
stages:
- stage: extract
job: read four documents into a strict schema
input_tokens: 6000
cost_of_error: low # the gate and verifier catch most misreads
latency_budget_s: 2
model_class: fast
- stage: gate
job: flag disputes where extracted fields disagree
input_tokens: 0
cost_of_error: medium # a miss sends a hard case down the fast path
latency_budget_s: 0.1
model_class: code
- stage: resolve_easy
job: propose a resolution when the documents agree
input_tokens: 900
cost_of_error: medium
latency_budget_s: 2
model_class: fast
- stage: reconcile
job: decide which evidence wins when documents conflict
input_tokens: 900
cost_of_error: high # a wrong credit goes to a customer
latency_budget_s: 60
model_class: reasoning
- stage: verify
job: check the arithmetic and that cited lines exist
input_tokens: 0
cost_of_error: high
latency_budget_s: 0.5
model_class: code
A stage earns reasoning only when the error cost is high, the input is under a couple of thousand tokens, and the latency budget is measured in tens of seconds. If only the first is true, shrink the input before reaching for the bigger model.
When nobody is waiting, batch it
If flagged disputes are worked the next morning, the reasoning stage doesn’t need to be online at all. OpenAI’s Batch API runs jobs asynchronously at half price, with results inside a day. A nightly batch of the flagged 27% halves what the reasoning stage costs and removes the p95 problem, since no request is sitting open.
The 27% belongs to Sedgemoor
The 27% came out of Sedgemoor’s mix of disputes. Your pipeline’s share of hard cases could be 5% or 60%, and at 60% the split buys little. The only way to know is to run all three variants over the same labeled set and read the table. If fast-only and reasoning-only land within a point or two of each other, the stage was never hard, and the extra thinking is buying latency.