An eval that reads only the final message will pass an agent that issued a refund first and checked the return window second. The customer was owed the money, the reply says so, and the score goes up. I wouldn’t accept that order of work from a person, and I don’t see why an agent should get a pass for it.
What follows is a small trajectory check: one YAML spec per scenario, and a Python script that scores a recorded tool-call trace against it. It runs beside the answer check and reports a second score.
What the recorder keeps
OpenAI function calling and Claude tool use both hand you the path for free. Each tool call arrives as a structured block with a name and JSON arguments, and the loop you own is what executes it. So the recorder is a wrapper around that loop. For every step it appends the tool name, the arguments, and the dollar cost of the model call that produced the step, worked out from the token usage on the response.
I write one JSON file per scenario run. If your traces already go to a store such as LangSmith, export from there and skip the wrapper. The checker needs only this much:
{
"scenario": "refund-inside-window",
"steps": [
{"tool": "get_order", "args": {"order_id": "Q-1042"}, "cost_usd": 0.004},
{"tool": "issue_refund", "args": {"order_id": "Q-1042"}, "cost_usd": 0.005},
{"tool": "check_return_window", "args": {"order_id": "Q-1042"}, "cost_usd": 0.004}
],
"answer": "Your refund of $18.50 is on its way."
}
That trace is the failure from the first paragraph. Nothing in the answer field gives it away.
One spec per scenario
scenario: refund-inside-window
must_call:
- get_order
- check_return_window
- issue_refund
must_not_call:
- escalate_to_human
order:
- before: get_order
after: check_return_window
- before: check_return_window
after: issue_refund
max_steps: 5
max_cost: 0.03
The order rules are precedence rules. The second one reads: every issue_refund needs a check_return_window somewhere before it. It says nothing about the two being adjacent. My first draft of this spec was the full expected sequence. I dropped it, because an exact-match golden path fails every time the agent does something harmless, like looking up the customer before the order. Precedence pairs let me pin the two or three orderings I care about and leave the rest alone.
must_not_call changes per scenario, which is why the spec lives with the scenario and not in one global file. For the order that’s outside its return window, issue_refund moves into that list.
The ceilings catch the agent that makes six calls where two would do. I set max_steps at about twice the path I’d take by hand. Set it tighter and the check fails on one extra lookup, and people learn to ignore it.
The checker
It needs PyYAML and nothing else. Run it as python check_path.py specs traces.
import json
import pathlib
import sys
import yaml
def check(spec, trace):
"""Return a list of failures. An empty list means the path passed."""
steps = trace["steps"]
calls = [step["tool"] for step in steps]
failures = []
for tool in spec.get("must_call", []):
if tool not in calls:
failures.append(f"missing required call: {tool}")
for tool in spec.get("must_not_call", []):
if tool in calls:
failures.append(f"forbidden call: {tool}")
for rule in spec.get("order", []):
first, later = rule["before"], rule["after"]
for i, tool in enumerate(calls):
if tool == later and first not in calls[:i]:
failures.append(f"{later} at step {i + 1} ran before any {first}")
break
max_steps = spec.get("max_steps")
if max_steps is not None and len(calls) > max_steps:
failures.append(f"{len(calls)} steps, ceiling is {max_steps}")
cost = sum(step.get("cost_usd", 0.0) for step in steps)
max_cost = spec.get("max_cost")
if max_cost is not None and cost > max_cost:
failures.append(f"cost ${cost:.4f}, ceiling is ${max_cost:.4f}")
return failures
def main(spec_dir, trace_dir):
spec_paths = sorted(pathlib.Path(spec_dir).glob("*.yaml"))
passed = 0
for spec_path in spec_paths:
spec = yaml.safe_load(spec_path.read_text(encoding="utf-8"))
trace_path = pathlib.Path(trace_dir) / f"{spec['scenario']}.json"
if not trace_path.exists():
print(f"FAIL {spec['scenario']}: no trace recorded")
continue
failures = check(spec, json.loads(trace_path.read_text(encoding="utf-8")))
if failures:
print(f"FAIL {spec['scenario']}: " + "; ".join(failures))
else:
passed += 1
print(f"PASS {spec['scenario']}")
print(f"{passed} of {len(spec_paths)} paths passed")
return 0 if spec_paths and passed == len(spec_paths) else 1
if __name__ == "__main__":
sys.exit(main(sys.argv[1], sys.argv[2]))
Against the trace above it prints FAIL refund-inside-window: issue_refund at step 2 ran before any check_return_window and exits non-zero, so it can gate a build. A missing trace counts as a failure. An agent that crashed before recording anything shouldn’t raise the pass rate by being absent.
Every failure is a plain sentence, and the ordering ones carry a step number. I care about that more than the score, because whoever reads the output is usually about to open the trace and wants to know where to look.
Pebblebank Books, scored both ways
Pebblebank Books is a shop I’m imagining, refund agent and all, and so are its numbers.
Say its refund agent runs 40 scenarios. The answer check, the kind of assertion on the final reply that promptfoo or OpenAI Evals handles well, passes 36 of them. Then the path check runs over the same traces. In 9 of those 36, the agent issued the refund before it checked the return window.
| Scored on | Passing | Score |
|---|---|---|
| Answer only | 36 of 40 | 90% |
| Answer and path | 27 of 40 | 67.5% |
Those nine came out right only because their scenarios put the order inside the window. Give the same habit an order from five months ago and money leaves that shouldn’t have. The 90% was partly a measure of how forgiving the scenario set was.
It also changes what you fix. Four wrong answers send you to the prompt wording. Nine refunds issued before the check send you to the tool design: maybe issue_refund should refuse to run without a window result passed in.
What I left for the judge model
Rules can’t tell me whether the reply explained a denied refund kindly, or whether a retry after a tool error was a reasonable thing to try. That’s judge-model work, and in the diagram the judge sits on the answer line, after the plain assertion, for the cases the assertion can’t settle.
I keep it off the path line on purpose. A rule costs nothing to run and gives the same verdict next month. A judge costs a model call per trace, and its verdicts can shift when the judge model changes underneath you. So the rules run first, and a trace that already failed on its path never reaches the judge. When the judge does run, it gets one narrow question and has to answer pass or fail.
What the checker doesn’t look at yet
Arguments are the obvious gap. The checker confirms issue_refund was called after the window check, and has no opinion on whether the amount matched the order. A small matcher on args is the next thing I’d add.
Per-tool call limits would help too. max_steps: 5 allows four order searches in a row.
And one run per scenario isn’t enough. The same agent on the same input can take a different path on the next run, so I’d run each scenario five times and report “path passed 3 of 5” per scenario. A single green tick hides the scenarios that need the most attention.
If you’re adding this to an existing eval set, write the first spec for a scenario that ends in a write. That’s where a bad path costs something.