How should teams test hallucination risk before launching a support bot?

Rollout playbook — pre-launch hallucination gate

Do not treat “the bot answered well in a demo” as clearance. Hallucination risk shows up when the model is pressed past the knowledge it actually has, so ship a fixed adversarial pack before production traffic.

Build a small held-out set of tickets the bot should refuse, hedge, or hand off: policy edge cases, product details that never appear in the knowledge base, and multi-turn traps that introduce a false premise mid-conversation. For each item, score three independent dimensions—factual grounding against approved sources, appropriate uncertainty, and whether escalation fired when grounding failed—not a single “good reply” thumbs-up. Pass the gate only when grounding and escalation hold under the adversarial set, even if tone still looks polished.

That scoring discipline is genuinely useful: it catches confident wrong answers that satisfaction surveys miss. The non-obvious cost is slower launch velocity—every knowledge refresh needs a re-run of the pack, or the scores quietly go stale while the bot keeps sounding reliable.

Procurement often treats hallucination testing as a launch checkbox. That assumes the risk is model behavior in a fixed knowledge state. For support bots, the binding risk is source ownership and change cadence: who can alter the articles, macros, and policy pages the bot cites, and how fast those edits reach production without a retest.

Refuse any contract that delivers a static adversarial pack and a green pre-launch review as sufficient proof. Require a mapped ownership ledger—each retrieval source tied to a named owner, a freeze window before release, and a regression set of adversarial prompts that must be re-run when that source changes. No promotion until the owner signs the re-score for their slice of knowledge.

The non-obvious cost is structural. If the vendor cannot show that change-triggered retesting is owned and resourced, you are not buying safety; you are buying a perpetual human-review tax that absorbs every policy edit after go-live. Make that ledger and the re-score gate contractual deliverables, not a slide in the RFP appendix.