Ask your bot an easy question about invoices. Then ask the same question as someone who can't access Billing. If it gives both customers the same steps, it isn't ready to answer that question in production. A good chatbot test checks not just what the bot says, but whose question it is answering and what should happen next.
1. Build a small test set from real chats
Choose ten recent questions from one workflow, such as billing or onboarding. Rewrite them to remove personal and account details, but keep the way people asked. Include a straightforward question, a vague version, a follow-up, a role or plan limit, a policy you do not document, and a direct request for a person.
Before running the test, write what a correct answer must contain, which article should support it, and when the bot must stop and hand off. Otherwise, a polished answer can pass even when it misses a critical detail.
2. Copy this QA sheet
| Test question | Test as | Expected answer or action | Source to verify |
|---|---|---|---|
| Where do I download last month's invoice? | Workspace owner | Give the current path and state the owner requirement. | The current invoice download article |
| I don't see Download. What now? | Member | Explain why and give the member a safe next step. | The permissions section of the same article |
| I was charged twice. Can you fix it? | Customer with a billing problem | Don't assume a refund or collect card details. Route to billing support. | Your approved handoff policy |
| Can I get an exception to the policy? | Signed-out visitor | Don't make up an exception. Explain what can be verified or hand off. | Public policy or no public source |
| No, I want to talk to a person. | Any test user | Stop repeating suggestions and route the request to a person. | Your escalation policy |
Add columns for actual reply, actual source, actual queue, pass or fail, and owner. Mark a response wrong if its source contradicts it, the link can't be opened by that audience, or the handoff goes to the wrong place.
3. Test in the tool you actually use
- Zendesk: follow Zendesk's AI agent testing guidance for your channel and setup. Use test identities to check the behavior in a customer-facing flow, not only in a builder preview.
- Intercom: use Fin Evals to repeat multi-turn scenarios when content, guidance, or workflows change. Manually review failures and high-risk passes, and verify the real Messenger handoff in a controlled test before opening it to customers.
Repeat critical tests with the intended brand, language, and user permissions. A correct answer for an admin is not a passing result for a signed-out customer. If the expected result is a handoff, confirm a teammate actually receives the conversation with enough context to continue.
4. Decide what blocks launch
Block launch for private content exposure, an incorrect billing or policy promise, or a broken required handoff. Fix the source article if it's wrong; fix source access or indexing if the right article isn't available; fix routing when the bot has the right words but takes the wrong action. Rerun the affected cases plus a few previously passing ones.
Start with a narrow audience and review early conversations manually. After launch, measure ticket deflection without counting abandoned chats. When the product changes, run the help center release checklist before rerunning your saved tests. If you still need to organize the source material, start with the AI knowledge base guide.