manage_order(action="cancel booking", date="next Friday", details="customer changed mind")
I’d fix the schema behind that call before touching the prompt or shopping for a better model. The tool asked for strings and explained none of them, so the model filled each one with plausible English. It followed the schema it was given.
Function calling is about a year old now. OpenAI shipped it last June, Gemini and Mistral have their own versions, and Anthropic’s tool use for Claude is in public beta. The wrappers differ a little. The part you write is the same everywhere: a name, a description, and a schema for the arguments, in JSON Schema or something close to it. The model reads all of it as instructions.
Five hundred calls from a booking assistant
The numbers here come from a company I invented, Kestrel Tickets. It sells event tickets, and its booking assistant has five broad tools. Someone on the team pulls 500 logged tool calls and labels each one by hand.
Seventy-one are wrong. That’s 14.2 percent, about one call in seven. In 33 the model picked the wrong tool. In 26 the arguments were invalid: free-text dates like the one above, and a seat class the venues have never sold. The last 12 left out a field the backend needed.
Each pile has a cause in the tool definitions. Five tools that each do several jobs overlap, and nothing says which to pick. Every argument is typed as a string, even where the allowed values were known. No field is marked required.
One tool hiding four operations
This is the definition behind that first call, in the tools format OpenAI’s API takes:
{
"type": "function",
"function": {
"name": "manage_order",
"description": "Manage a customer's order.",
"parameters": {
"type": "object",
"properties": {
"action": { "type": "string" },
"order": { "type": "string" },
"date": { "type": "string" },
"details": { "type": "string" }
}
}
}
}
The description says what the tool is and nothing about when to call it. action hides lookup, change, cancel and refund, each needing different arguments, so none of them can be required. That’s how the call at the top got through with no order number in it. And a cancellation, which moves money, sits behind the same name as a harmless read.
Two tools with less room to guess
Lookup and cancel become their own tools. Changes and refunds outside a cancellation get the same treatment, and I’ve left them out to keep this short.
[
{
"type": "function",
"function": {
"name": "get_order",
"description": "Look up one existing order by its order number. Returns status, event name, event date and seats. Use it when a customer asks about an order, and always before cancel_order. Do not use it to search for events or to check seat availability.",
"parameters": {
"type": "object",
"properties": {
"order_id": {
"type": "string",
"pattern": "^KT-[0-9]{8}$",
"description": "Order number from the confirmation email, for example KT-20481137."
}
},
"required": ["order_id"],
"additionalProperties": false
}
}
},
{
"type": "function",
"function": {
"name": "cancel_order",
"description": "Cancel a whole order and start its refund. Use it only after get_order has returned the order and the customer has confirmed the cancellation. Do not use it to change seats or dates, or to cancel one ticket out of several.",
"parameters": {
"type": "object",
"properties": {
"order_id": {
"type": "string",
"pattern": "^KT-[0-9]{8}$",
"description": "Order number, exactly as get_order returned it."
},
"event_date": {
"type": "string",
"format": "date",
"description": "Event date on the order as YYYY-MM-DD, for example 2024-06-21. Copy it from get_order."
},
"reason": {
"type": "string",
"enum": ["customer_request", "event_cancelled", "event_rescheduled", "duplicate_order", "payment_problem"],
"description": "Closest match to why the order is being cancelled."
}
},
"required": ["order_id", "event_date", "reason"],
"additionalProperties": false
}
}
}
]
event_date is there on purpose. A cancel has to repeat the date get_order returned, so a mistyped order number fails a check before it cancels someone else’s tickets.
The schema is still only a request. A model can send back arguments that break it, so pattern, enum and required help the model aim and something else has to check.
Eight checks for every tool definition
- The tool does one job. If the name needs “manage” or “handle”, split it, and keep reads apart from writes.
- The description says when to use the tool and when not to, and names the tool to use instead.
- Any argument with a known set of values is an
enum, cancellation reasons and seat classes included. - Dates and IDs carry a format, with an example in the description.
- Everything the backend needs is in
required. Optional fields say what happens when they’re left out. - There are no catch-all arguments like
detailsoroptions, andadditionalPropertiesis false. - Argument names say what they hold:
order_idandevent_date, neverorderanddate. - Arguments are validated before anything executes, and a failure goes back to the model as a message it can act on.
What the validator says when it rejects a call
The validator sits between the tool call and the executor. Both it and the model read one registry of definitions, so a call is checked against the schema the model saw. JSON Schema validation runs first, then the checks a schema can’t express, such as whether event_date matches the order. One catch: most validators ignore format unless you switch format checking on, so test that a bad date really fails. It’s the same gate I’d put in front of SQL a model wrote: check it before it runs, and send the reason back when it fails.
A failed call comes back to the model as a tool result instead of an exception, and the wording of that result decides whether the retry works:
{
"error": "invalid_arguments",
"tool": "cancel_order",
"problems": [
{
"field": "event_date",
"received": "next Friday",
"expected": "YYYY-MM-DD, copied from get_order"
},
{
"field": "reason",
"received": "changed mind",
"allowed": ["customer_request", "event_cancelled", "event_rescheduled", "duplicate_order", "payment_problem"]
}
],
"next_step": "Nothing was cancelled. Fix these fields and call cancel_order again."
}
“Invalid input” would have earned the same bad call again. Each problem here names the field and sets what arrived next to what’s allowed, and that’s enough for the model to correct itself on the next turn. I cap it at one retry, then have the assistant ask the customer. In Python, Instructor will run this validate-and-retry loop for you with Pydantic models.
Two smaller stops sit after the executor. A result shaper returns the handful of fields the model needs instead of the backend’s whole payload, in one consistent form for successes and failures. And every call is logged with its arguments and the validator’s verdict, retries included.
Nine tools, 13 misses, and a longer request
Kestrel splits its five tools into nine, with enums wherever the values were known, and turns on the validator messages. In the next 500 calls, 13 are wrong on the first attempt, or 2.6 percent. The validator rejects each of them, and all 13 come back correct after one retry. A wrong pick among narrow tools tends to trip the validator too, because the arguments for one job rarely fit another. These numbers are invented. They show how to keep score, and a real log would be messier.
The split isn’t free. Tool definitions ride along with every request, so nine of them cost more tokens than five. A cancellation now takes two calls where it took one, and I’m happy to pay that latency for anything that writes. Enums go stale: the day a venue adds a seat class, the schema is wrong until someone updates it, so I’d generate the enum from the same table the backend reads. And nine tools with clear edges are easier to choose among than five blurry ones, but I wouldn’t assume that still holds at ninety.
The call log is the piece I’d build before the first narrow tool. Without it, Kestrel would have had a feeling about its tools and no piles to count.