Someone asks a chatbot how to download an invoice. The bot replies, and the customer leaves. Maybe they got the invoice. Maybe they gave up and emailed support. Silence isn't enough to tell those two outcomes apart.
Start with one measurable question: Of the eligible support attempts we observed, how many were resolved without a person and didn't return as the same issue? That's a more conservative measure than a dashboard count of chats closed without handoff.
1. Define what counts as an attempt
Pick one workflow, such as invoice downloads. Count only sessions in which someone asked a question or entered an identifiable self-service flow for that issue. Exclude spam, test sessions, and widget opens with no question or task signal. Decide what an “eligible attempt” is before comparing results.
Choose a follow-up window that fits the task, such as seven days. Where your data allows, match later contacts on the same issue across chat, tickets, and email using an approved account or conversation identifier. Report match coverage. Don't assume customers with unlinked identities never came back.
2. Classify the outcomes
| Outcome | Evidence | Count as verified deflection? |
|---|---|---|
| Resolved without a person | Customer confirmed the issue was solved or the intended task succeeded; no observed same-issue follow-up where matching is available. | Yes |
| Handed off | A teammate took over or a ticket was created for the issue. | No |
| Repeat contact | Customer returned about the same issue within the follow-up window. | No |
| Abandoned or unknown | Customer went quiet, dropped before getting help, or there isn't enough evidence to verify the outcome. | No |
Use one final outcome per eligible attempt. A later same-issue ticket overrides an initial confirmation. If you know a handoff happened, record it as a handoff even if the customer also stopped chatting.
3. Keep the denominator visible
Verified self-service resolution rate = verified resolutions without a person ÷ eligible support attempts. For example, 100 eligible attempts produce 38 verified resolutions, 25 handoffs, 12 repeat contacts, and 25 unknowns. Report 38% verified, not 63% by adding the unknowns.
This is an observed self-service outcome, not proof that exactly 38 tickets were prevented. To estimate avoided tickets, compare human contacts per 1,000 eligible accounts or task attempts with a comparable baseline or a holdout. Keep the topic, customer mix, channels, and time window consistent. Also track repeat contact and customer satisfaction; a lower ticket count is not a win if more people leave frustrated.
4. Read vendor metrics alongside yours
Intercom's reporting definitions include both confirmed and assumed resolutions in Fin's resolution rate. Zendesk has its own AI agent conversation statuses. Use those dashboards to diagnose trends, but don't compare their totals as if the definitions were identical. Keep the confirmed, assumed, and unknown counts separate wherever your data supports it.
Review a sample of unknowns each week. If people leave after an unclear permission answer, fix the article and rerun the chatbot test. If your team is deciding which issues to automate first, use the customer support automation guide.