Sooner or later, every support chatbot gets a question its documentation doesn’t cover. A well-built one says it doesn’t know and offers to connect the customer with a person. A badly built one answers anyway, fluently and with total confidence.
That second behavior is what I worry about most when an LLM goes on a public website. Anyone on the internet can type into a support widget, including people who are actively trying to break it, and every answer goes out under your company’s name. I was tech lead for the customer-support GenAI agent at Okta, an identity and security company, where a confident wrong answer about configuring sign-in is worse than no answer at all.
So the design below is mostly about one thing: answer when there are grounds to, and get out of the way when there aren’t.
Two checks around one model call
- The customer types a question into the chat widget.
- Before any model sees it, the input gets screened. Is it in scope? Is it an attempt to manipulate the agent? Does it contain data we shouldn’t store?
- The system searches approved sources, mainly documentation and knowledge-base articles, for passages that bear on the question.
- The model drafts an answer from those passages only, and links to them.
- The draft is checked before it’s sent. Every claim has to trace back to a retrieved passage, and the answer has to stay inside policy.
- If it passes, the customer gets it. If it doesn’t, a person gets the conversation, with the history attached.
Teams in a hurry cut steps two and five first. Without them, I wouldn’t put this on a public website.
Bad answers usually start in search
When an answer is wrong, my first suspect is retrieval. Most of the time the model did a reasonable job with what it was handed. It was just handed the wrong passages.
So I’d put the early effort into search. Clean up the sources. Chunk them so each passage carries enough context to stand on its own. Mix keyword and semantic matching, because error codes and setting names are exactly where pure embeddings get fuzzy.
And measure retrieval by itself, apart from the final answer. If you only score end to end, you can’t tell a search problem from a prompt problem, and you’ll spend a week tuning the wrong one.
Index only what you’d say to a customer
The agent knows what’s in its index and nothing else. I think of that as a feature. Put only content that’s approved for the audience in there.
If public and internal content share an index, someone will eventually phrase a question that pulls up an internal page. A filter at the end of the pipeline is a weak promise to make. Keep internal content out of the index entirely. You can’t leak what you never indexed.
Text that arrives at run time doesn’t get a vote
A customer message can contain text written to redirect the agent. So can a retrieved page or a pasted log file. I treat all of it as material to read and never as instructions to follow. The rules live in the system prompt and, more importantly, in the code around it. A model can still be talked out of an instruction, so anything that must hold, like which sources it can quote or which actions it can take, is enforced in code that no user text or document can reach.
Screening catches the obvious attempts. The design has to hold up against the ones it misses, which is why the check in step five looks at the output as well.
The agent borrows the customer’s identity
Once the agent can do more than read docs, say look up a support case or add a note to one, it should make that call as the signed-in customer, with that customer’s permissions. Never through a service account that can see everyone’s cases.
The agent never holds more access than the person it’s talking to. Then if an injection does get through, the most it can reach is what that customer could already see or do. That’s also why anything that changes data should ask the customer to confirm first.
Design the “I’m not sure” answer
Decide up front what the agent does when it can’t ground an answer, and make handing off to a person a first-class outcome instead of an error path. When it hands off, attach the conversation. Nobody wants to explain the problem twice.
Track the handoff rate. Just don’t treat it as a number to drive to zero. A falling handoff rate is good news only if the answers stayed correct. Push it down by making the agent bolder and you get the confident wrong answer from the top of this post.
Before launch: evals and a real security review
I’d build the evaluation set before the agent. Collect real customer questions with known good answers, then add adversarial ones, like injection attempts and questions the docs simply don’t answer. Every prompt change and every model upgrade runs against that set before it ships. New sources too, since one new document can shift what retrieval returns for questions you thought were settled. Without the set, you’re judging quality by whichever conversation someone screenshotted last.
The agent should also go through security review like any other public service. Threat-model it and classify the data it touches, then decide what gets logged and how long you keep it. An AI feature doesn’t get a lighter process because it’s new.
Do those reviews early. Done before the build, they change the design. Done after, they mostly produce a list of things you wish you’d done.
What I’d watch once it’s live
The headline number is the share of questions resolved without a person, along with how satisfied customers are with those answers. I’d put it right next to the share of answers where every claim ties back to a source. If the first climbs while the second slips, the agent is guessing more.
The report I’d read every week is the list of questions it couldn’t answer, grouped by topic. Each cluster is a gap in your documentation, described in your customers’ own words, and your docs team will rarely get a better input than that.