Keeping LLM costs predictable as usage grows

The LLM bill that gets a meeting arrives months after launch. Pricing every call in one place, before the provider sees it, keeps that bill from being a surprise.

Request Cache Router Small model Large model Budget guard Provider Usage log hits return here daily cost report
Cache first, then the router picks the smallest model that can do the job, and a budget check runs before the provider sees the request.
On this page
  1. What every call passes through
  2. The cache is the cheapest model you’ll ever call
  3. Route on what you know before the call
  4. Why the guard sits after the router
  5. The math for one feature
  6. The daily report people actually open
  7. What this setup won’t do for you

The first LLM bill rarely worries anyone. It’s small, and the feature works. The bill that gets a meeting on the calendar comes a few months later, when usage has grown the way everyone hoped and cost has grown in ways nobody planned.

Predictable spend comes from plain plumbing: every request takes the same path, and every stop on it knows what it costs.

What every call passes through

All model traffic goes through one internal service, and feature teams don’t hold provider keys. That rule does more for cost than anything clever further down, because it gives you one place to cache, route, count, and say no. It’s the same gateway as in the post on putting an LLM inside the CRM, seen from the bill’s side.

In order:

  1. The response cache. If this exact request was answered recently, return the stored answer.
  2. The router. It picks a small model or a large one.
  3. The budget guard. It prices that choice against the feature’s budget and the tenant’s.
  4. The provider, with a hard cap on output tokens.
  5. The usage log. One row per request, cache hits included, tagged with feature, tenant, model, tokens, and cost. The daily cost report reads from here and nowhere else.

The cache is the cheapest model you’ll ever call

A cache hit costs next to nothing. It pays off when people keep asking for the same thing, like agents reopening a ticket that hasn’t changed. For open-ended chat the hit rate is near zero, and I wouldn’t bother.

A provider’s prompt caching is a separate thing: it discounts a repeated prefix but still runs the model.

Semantic caching is where I get wary, at least for anything a user reads. “Similar enough” sounds efficient until you serve someone the answer to a question they didn’t quite ask. Exact matching on a normalized input is less impressive and much safer.

Whatever library you use, put the tenant in the key, or the cache becomes a very fast way to leak one customer’s data to another. And store only answers that passed validation, or one bad answer gets served all day.

Route on what you know before the call

The signals worth routing on are the obvious ones: which feature is calling, how long the input is, whether the output has a format you can check.

Don’t call the large model to find out whether you need the large model.

Escalation is the other half. The small model goes first, its output is validated, and only a failure moves up. That works when validation is real, like a JSON schema, and badly when “failure” is a judgment call. Either way, an escalated request pays for both calls, and the second one passes the guard too.

Here is a policy small enough to review in a pull request:

# routing-policy.yaml
# Illustrative. Model ids and prices are placeholders, not any provider's price list.
version: 7

models:
  small:
    id: small-model-v2
    usd_per_million_tokens: { input: 0.20, output: 0.80 }
    max_output_tokens: 500
  large:
    id: large-model-v4
    usd_per_million_tokens: { input: 3.00, output: 15.00 }
    max_output_tokens: 1000

cache:
  match: exact
  key: [tenant, feature, template_version, normalized_input]
  store_only_validated: true
  ttl_hours: 24

features:
  ticket_summary:
    default_tier: small
    large_when:
      input_tokens_over: 6000
    escalate_on: [schema_invalid, missing_required_field]
    escalation_alert_share: 0.15
    monthly_budget_usd: 3600
  contract_clause_review:
    default_tier: large
    monthly_budget_usd: 2500
    on_soft_limit: alert_only   # a weaker contract review isn't worth the savings
  reply_draft:
    default_tier: small
    escalate_on: []
    monthly_budget_usd: 800

tenants:
  default_daily_usd: 25
  overrides:
    tenant_0042: 60

guard:
  estimate_output_tokens: max_output_tokens
  on_pace_over: alert_owner
  soft_limit_percent: 80
  on_soft_limit: downgrade_to_small
  hard_limit_percent: 100
  on_hard_limit: refuse_with_message

Output is capped because output tokens cost four or five times as much as input here, and a cap bounds any single call. The escalation_alert_share line is there because a rising escalation rate is often the first sign that a prompt or a model changed under you.

Why the guard sits after the router

A guard in front of the router can only ask whether a tenant is over budget. Behind it, the guard can ask whether this particular call is affordable, and it has a cheaper option ready: the small model.

It prices each call from counted input tokens plus the full output cap. That overestimates, sometimes by more than double, which I’m fine with before the money is spent. The running totals come from the usage log, so they’re real. Per-feature budgets stop one product area from spending another’s money. Per-tenant caps stop one customer’s runaway script from spending everyone’s.

I also check pace. A feature that has used 60% of its month by the 8th is a problem today, even while it’s under the limit.

The hard limit is the part I’m least sure about, and I don’t think it’s purely an engineering call. Refusing with a clear message is honest. Quietly degrading is friendlier and harder to notice. I lean toward the message plus an alert to the owner, but I can see letting a paying tenant run over and settling it in the contract.

The math for one feature

The prices come from the policy above, and the traffic numbers are round.

Say the ticket summary feature handles 40,000 requests a day, 1.2 million in a 30-day month. Most tickets are short, about 1,000 input tokens and 250 output. One in twenty is long, about 8,000 in and 400 out, which trips the 6,000-token rule.

small, short ticket = 1,000 x $0.20 / 1M + 250 x $0.80 / 1M  = $0.00020 + $0.00020 = $0.00040
large, short ticket = 1,000 x $3.00 / 1M + 250 x $15.00 / 1M = $0.00300 + $0.00375 = $0.00675
large, long ticket  = 8,000 x $3.00 / 1M + 400 x $15.00 / 1M = $0.02400 + $0.00600 = $0.03000

baseline, everything on the large model, no cache:
  1,140,000 short x $0.00675 + 60,000 long x $0.03000 = $7,695 + $1,800 = $9,495 a month

For the same short ticket, the large model costs almost 17 times as much. Now apply the policy with a 20% cache hit rate, spread evenly, and 10% of small calls escalating:

Line Count per month Cost each Monthly cost
Cache hits (20% of requests) 240,000 $0 $0.00
Small model, short tickets 912,000 $0.00040 $364.80
Large model, long tickets 48,000 $0.03000 $1,440.00
Large model, escalated (10% of small calls) 91,200 $0.00675 $615.60
Policy total 1,200,000 requests $2,420.40

That’s about a quarter of the baseline. Cache hits show $0 because I’m only counting model spend. Escalated calls already paid for their small attempt in the small-model row.

The row I’d stare at is the long tickets: 5% of requests, about 60% of the bill. The large model as a whole is about 85% of it, and the small model, which handles most of the calls, is about 15%. If I had to cut further, I’d strip quoted email history and signatures from long tickets before shopping for a cheaper small model.

Then the budget. The policy gives this feature $3,600 a month, a pace of $120 a day, and it normally runs at about $81. Now say a prompt change trips up the small model and 40% of its summaries start failing the schema check. Each failure escalates, and daily spend jumps to about $142. The 15% escalation alert and the pace check both fire that day, weeks before an invoice would. Left alone from the first of the month, the feature hits its 80% soft limit around day 20.

The daily report people actually open

The report is a query over the usage log, sent each morning to each feature’s owner: spend by feature and tenant, model mix, cache hit rate, escalation rate, and the change since yesterday. Owners see their own line, which is what makes a budget feel real.

I’d have the gateway refuse any call without a feature tag and a tenant tag. Untagged spend piles up in a row called “other”, and nobody owns “other”.

What this setup won’t do for you

Routing rules drift. When a provider ships a better small model, your thresholds are suddenly wrong, sometimes in your favor. I’d rerun the evaluation set before every model switch.

And none of this asks whether a request needs a model at all. Plenty could be answered by a lookup or a plain rule, and design review, before the traffic exists, is the place to catch those.