Nine labels don't need a frontier model

A build log for email triage: lock a 3.8B model to a fixed label set, pick a confidence threshold from labelled data, and escalate only the uncertain messages.

Message Small model Confidence Threshold gate Large model Router Disagreement log 9 labels only accept at 0.80+ fallback below 0.80 weekly audit
A small model labels each message, a threshold gate sends low-confidence ones to a large model, and both paths meet at the router.
On this page
  1. Log 1: the labelled sample comes before the prompt
  2. Log 2: one token per label, and nothing else allowed
  3. Log 3: the number 0.80 comes out of a table
  4. Log 4: what a day of mail looks like
  5. Log 5: the weekly audit rule

Entry one in this build log is a spreadsheet: 2,000 inbound emails, each with a label a person agreed on. No prompt exists yet. No model has been picked either.

I start there because a lot of the traffic in an LLM application is classification. Which queue does this email go to? Is it spam? Once the label set is fixed, a model with a few billion parameters handles most of that as well as the biggest one you can rent. “Most” is a number you can measure, and the rest of this log is about measuring it.

The company is one I invented for the exercise, and so are its numbers. Farrowdale Utilities gets 46,000 customer emails a day and sorts them into nine queues. Today every email goes to a large hosted model.

Log 1: the labelled sample comes before the prompt

The 2,000 messages are pulled at random from a month of mail, so rare labels show up at their real rate. Two people label each one, and the messages they disagree on get argued out. That argument is useful by itself, because it usually ends with a label definition getting rewritten.

I hold the sample fixed from here on. Every model and prompt change gets scored against the same 2,000.

Log 2: one token per label, and nothing else allowed

The prompt gives each label a letter and asks for the letter.

You sort customer emails for a utility company.
Reply with one letter and nothing else.

A  Billing dispute or bill question
B  Payment plan or trouble paying
C  Power outage report
D  Start, stop or move service
E  Meter reading or meter fault
F  Gas smell, downed line or other safety hazard
G  Complaint about staff or service
H  Spam or vendor solicitation
I  Anything else

Email:
{email}

Letter:

Asking nicely isn’t a constraint, though. A small model will sometimes answer “The letter is A” or invent a tenth category. So I don’t let it generate at all. I read the scores for the next token, keep only the nine letters, and normalize those into probabilities. The top one is the label and its probability is the confidence.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "microsoft/Phi-3-mini-4k-instruct"
CODES = "ABCDEFGHI"
PROMPT = open("triage_prompt.txt").read()

tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
    MODEL, torch_dtype=torch.float16, device_map="auto"
)
code_ids = [tok.encode(c, add_special_tokens=False)[-1] for c in CODES]

def classify(email_text):
    messages = [{"role": "user", "content": PROMPT.format(email=email_text)}]
    ids = tok.apply_chat_template(
        messages, add_generation_prompt=True, return_tensors="pt"
    ).to(model.device)
    with torch.no_grad():
        logits = model(ids).logits[0, -1]
    probs = torch.softmax(logits[code_ids].float(), dim=-1)
    best = int(probs.argmax())
    return CODES[best], float(probs[best])

I haven’t run this exact snippet against a GPU for this post, so treat it as a starting point and check two things first. Each letter has to map to a single token, and it has to be the variant the model emits at that position: some tokenizers have one “A” with a leading space and one without. Letters that are also words, A and I here, can pick up stray score too. If “Anything else” looks inflated, switch the codes to digits. If your labels have to be multi-token strings or JSON, a constrained generation library such as Outlines does the same job with outlines.generate.choice, though you’ll need another route to a probability. I’d use Ollama to try candidate models quickly, and move to code like the above for the scored runs, since I want the raw token scores.

In the Farrowdale exercise, Phi-3-mini at 3.8B parameters gets 1,828 of 2,000 right, which is 91.4 percent. That’s a figure I chose for the example, not a benchmark result. Mistral 7B and Llama 3 8B are worth running through the same sample. I’d take the smallest model that clears the bar, because latency and memory are paid on every message.

Log 3: the number 0.80 comes out of a table

91.4 percent isn’t good enough to route mail with. About one email in twelve would land in the wrong queue.

The errors aren’t spread evenly, though. They cluster where the model was unsure. So I sort the 2,000 results by confidence and compute, for each candidate threshold, how much mail the small model would keep and how often it would be right on what it kept.

Threshold Share accepted Precision on accepted Escalation rate
none 100% 91.4% 0%
0.60 94% 94.3% 6%
0.70 89% 96.1% 11%
0.80 83% 97.6% 17%
0.90 71% 98.7% 29%
0.95 58% 99.1% 42%

Farrowdale picks 0.80. The small model keeps 1,660 messages and is right on 1,620 of them. The other 340 go to the large model.

Two cautions about this table. The confidence is a softmax over nine token scores, which is a ranking signal and not a calibrated probability. A score of 0.80 does not mean 80 percent correct, and that’s exactly why the threshold is read off labelled data. Also, the rows near the bottom rest on few errors. At 0.95 the precision figure comes from ten wrong answers, so I wouldn’t argue over a tenth of a point there.

I also run the table per label. An overall 97.6 percent can hide one label sitting at 85. And label F gets its own rule: anything the small model scores as a safety hazard, at any confidence, goes to the large model and a human queue. A missed gas leak costs far more than a misrouted billing question, and one global threshold can’t express that.

Log 4: what a day of mail looks like

At 46,000 messages a day, 38,180 are labelled locally at about 90 ms each on a single GPU. The remaining 7,820 go to the large model, which now sees 17 percent of the traffic it used to.

The saving has a price. At 97.6 percent precision, roughly 900 of those accepted messages land in the wrong queue every day. Whether that’s acceptable depends on what a wrong queue costs: a re-route by an agent, in Farrowdale’s case. It also depends on how the large model scores on the same 2,000, so measure that too. If it only gets 96 percent right, the split is an improvement. The large model tends to get treated as the answer key, and it’s just another classifier with an error rate.

Log 5: the weekly audit rule

A threshold picked in July describes July’s mail. A new tariff appears and customers start writing about things the sample never contained. So the router writes a log, and there’s a standing rule for reading it.

Weekly audit

1. Sample 1 percent of the labels the small model accepted.
   Send each one to the large model as well.
2. Log every disagreement: the sampled ones, plus every escalated
   message where the large model's label differs from the small
   model's top label.
3. A person reads all disagreements from the 1 percent sample and
   records which model was right.
4. Count escalation disagreements by label pair (small -> large).
   Read the ten largest pairs.
5. If precision on the sample drops below the threshold table's
   figure by more than one point, re-label 500 fresh messages
   and rebuild the table.

The disagreement log is also where new training data comes from. If one label pair keeps showing up, the first move is rewriting those two label definitions in the prompt. If that doesn’t hold, the fine-tune or retrieve question applies, and a few hundred logged disagreements make a decent start on a tuning set.

I’d rerun the full 2,000 against any new small model before swapping it in. The threshold belongs to one model and one prompt, and it doesn’t transfer.