A strict schema also decides the order the model writes in

Strict JSON schemas end unparseable replies, but key order shapes the answer, so put evidence and reasoning before the verdict and check meaning after parsing.

Document Schema Model call Parsed object Validator Typed record Schema eval Repair call evidence first labeled letters valid JSON, wrong facts one retry only
A schema-constrained call feeds a validator with one repair loop, and an eval compares schema versions on the labeled letters.
On this page
  1. One field drags the score down
  2. The schema that asks for the verdict first
  3. The same fields with the evidence ahead of the decision
  4. Six rules I’d apply to any extraction schema
  5. Checks that run after the parse

Since August 6 I’ve had no good reason to write a retry loop for broken JSON. OpenAI’s Structured Outputs takes a JSON Schema with strict: true and constrains decoding so the reply always conforms to it. JSON mode, from last November, only promised that the reply would parse. Outlines has done constrained decoding on open models since 2023, and Gemini 1.5 Pro accepts a response schema. Instructor works from the other end: validate with Pydantic, ask again on failure.

So the reply that doesn’t parse is nearly finished as a class of bug. Two cases survive. The model can refuse, which shows up in a separate refusal field, and a reply that hits the token limit still stops mid-object. Both are flagged on the response, so I check for them and skip the guessing. I’m glad to see the rest go.

The side effect gets less attention. Under a strict schema the model writes the keys in the order the schema lists them. That makes the schema a script for the order the model works in, and most extraction schemas I see are ordered like the database table they feed, with the status column near the top.

One field drags the score down

To show the effect I’ll use an insurer that doesn’t exist, Alderbank Claims, with letters and scores I made up.

Alderbank pulls six fields from each incoming claim letter: claim number, claimant name, letter date, loss date, claimed amount, and a liability decision, meaning whether the letter accepts, denies or partly accepts liability. Someone labels 400 letters by hand, and the same model, gpt-4o-2024-08-06, runs over them three ways.

Setup Unparseable replies Field accuracy Liability decision correct
JSON mode, schema described in the prompt 14 of 400 (3.5%) 92.1% of parsed fields 316 of 386 (81.9%)
Strict schema, verdict first 0 90.3% 285 of 400 (71.3%)
Strict schema, evidence first 0 94.6% 370 of 400 (92.5%)

The first row is how this worked until last month. Fourteen replies came back with a cut-off brace or a sentence of prose wrapped around the object. Strict mode fixes all fourteen, and the overall score still drops.

The liability field accounts for all of it. Dates and claim numbers get copied off the page, so their position in the schema hardly matters. Liability has to be inferred from sentences like “without admission of fault, we are prepared to offer”, and the verdict-first schema asks for it before anything else has been written.

The schema that asks for the verdict first

{
  "type": "object",
  "properties": {
    "liability_decision": { "type": "string", "enum": ["accepted", "denied", "partial"] },
    "reason": { "type": "string" },
    "claim_number": { "type": ["string", "null"] },
    "claimant_name": { "type": "string" },
    "letter_date": { "type": "string", "description": "ISO 8601 date, YYYY-MM-DD" },
    "loss_date": { "type": ["string", "null"], "description": "ISO 8601 date, YYYY-MM-DD" },
    "claimed_amount": { "type": ["number", "null"] }
  },
  "required": ["liability_decision", "reason", "claim_number", "claimant_name",
               "letter_date", "loss_date", "claimed_amount"],
  "additionalProperties": false
}

It’s a valid schema, and it looks tidy. But reason comes after the enum, so by the time the model writes it the decision is already on the page. All the field can do is defend that decision. The enum also has three values, so a letter that only acknowledges receipt of the claim gets pushed into one of them.

The same fields with the evidence ahead of the decision

{
  "type": "object",
  "properties": {
    "claim_number": { "type": ["string", "null"] },
    "claimant_name": { "type": "string" },
    "letter_date": { "type": "string", "description": "ISO 8601 date, YYYY-MM-DD" },
    "loss_date": { "type": ["string", "null"], "description": "ISO 8601 date, YYYY-MM-DD" },
    "claimed_amount": { "type": ["number", "null"] },
    "evidence_quote": {
      "type": "string",
      "description": "The sentence from the letter that best shows the liability position, copied word for word. Empty string if there is none."
    },
    "reasoning": {
      "type": "string",
      "description": "Two sentences at most on what the quote implies about liability."
    },
    "liability_decision": {
      "type": "string",
      "enum": ["accepted", "denied", "partial", "unknown"],
      "description": "Use unknown when the letter takes no position."
    }
  },
  "required": ["claim_number", "claimant_name", "letter_date", "loss_date",
               "claimed_amount", "evidence_quote", "reasoning", "liability_decision"],
  "additionalProperties": false
}

Either one goes into response_format as type json_schema, with a name and strict: true. Strict mode wants every property listed in required and additionalProperties set to false, which is why optional fields are written as a union with null. It doesn’t support format or pattern yet, so a date is just a string with a hopeful description.

The second version makes the model copy a sentence, say what it implies, and only then choose. That’s the old “think before you answer” prompt, moved into the key order.

Six rules I’d apply to any extraction schema

  1. Order the properties the way you’d want the work done: copied facts, then evidence, then reasoning, then the decision.
  2. Ask for a word-for-word quote ahead of any judgment field. A quote can be checked by code, and a paraphrase can’t.
  3. Give every enum an unknown and every field that may be missing a null. A constrained model can’t skip a field, so without those it has to make something up.
  4. Keep the reasoning field short and say so in its description. Those tokens cost money and latency on every call.
  5. Write descriptions as instructions, because the model reads them that way. I went into this for tool arguments in a wrong tool call is usually a schema bug.
  6. Version the schema and rerun the labeled set on every change. Reordering two keys is a prompt change, even if the diff looks cosmetic.

Checks that run after the parse

In the evidence-first run all 400 replies parsed, and nine of them had a loss date later than the date on the letter. A letter can’t report a loss that hasn’t happened yet. The schema has no way to say that, so plain code has to.

import re
from datetime import date


def squash(text: str) -> str:
    return " ".join(text.split())


def semantic_errors(rec: dict, letter_text: str) -> list[str]:
    errors = []

    dates = {}
    for key in ("letter_date", "loss_date"):
        if rec[key] is None:
            continue
        try:
            dates[key] = date.fromisoformat(rec[key])
        except ValueError:
            errors.append(f"{key} is not a YYYY-MM-DD date: {rec[key]!r}")

    if len(dates) == 2 and dates["loss_date"] > dates["letter_date"]:
        errors.append("loss_date is after letter_date")

    quote = squash(rec["evidence_quote"])
    if quote and quote not in squash(letter_text):
        errors.append("evidence_quote does not appear in the letter")
    if rec["liability_decision"] != "unknown" and not quote:
        errors.append("a decision other than unknown needs an evidence_quote")

    if rec["claimed_amount"] is not None and rec["claimed_amount"] < 0:
        errors.append("claimed_amount is negative")
    if rec["claim_number"] and not re.fullmatch(r"ALD-\d{7}", rec["claim_number"]):
        errors.append("claim_number does not match ALD-0000000")

    return errors

A record with errors goes back to the model once, with the letter and the error list attached. If the repair fails as well, a person gets it. The loop is short enough to write by hand. I’d cap it at one retry, because a second repair call rarely knows anything the first one didn’t.

I’d store evidence_quote on the final record too. When an adjuster questions a decision six months later, the sentence the model relied on is sitting right next to it.