Twelve metrics worth your attention — each with what it measures, what good looks like, and what to do when it moves the wrong way — plus five vanity metrics you can stop checking.
The 12 metrics that actually matter
1. True resolution rate
What it measures: Conversations fully resolved — no repeat contact about the same issue within seven days, no escalation.
What good looks like: 70–85% on routine intents (order status, shipping, returns). Lower is normal for complex topics.
How to act on it: Rank intents by failure volume, not failure rate. Retrain the top five offenders first.
2. CSAT, split by handler
What it measures: Satisfaction separately for bot-resolved and human-resolved conversations. The split is the point — a blended score hides everything.
What good looks like: Bot CSAT within about five points of your human baseline.
How to act on it: Treat the gap as a gate. If it trails badly, stop expanding automation and retrain.
3. Escalation rate and median time-to-escalation
What it measures: What share of conversations reach a human, and how long the bot wrestles before admitting defeat.
What good looks like: 15–30% escalation is healthy — zero usually means the bot traps people. Median time-to-escalation under 60 seconds.
How to act on it: Tighten confidence thresholds so uncertainty triggers a fast handoff with full context attached. A bot that argues for six minutes is worse than one that hands off in thirty seconds.
4. Chat-engaged conversion rate vs. unassisted
What it measures: Purchase rate of sessions with chat engagement versus sessions without.
What good looks like: 1.2x to 2x your site average. Below 1x means chat is hurting — usually badly timed proactive triggers.
How to act on it: Segment by trigger type. If proactive chat converts worse than reactive, your triggers interrupt instead of assist. Fix timing and copy first.
5. Revenue per chat-engaged session
What it measures: Revenue from chat-engaged sessions divided by their count. Conversion rate tells you how often; this tells you how much.
What good looks like: A rising trend quarter over quarter. Benchmark against yourself, not the industry.
How to act on it: Break it down by page and conversation type. Double down on what lifts it; rework or retire what doesn’t.
6. Cost per resolved conversation
What it measures: Fully loaded support cost divided by truly resolved conversations, blended across human and AI handling.
What good looks like: Falling steadily as automation absorbs routine volume — without the CSAT split deteriorating.
How to act on it: Compare against your pre-bot baseline. If it isn’t dropping after 90 days, your automation scope is wrong — you’re automating loud complaints instead of automatable intents.
7. Containment, done right
What it measures: Share of conversations resolved without human touch — with a quality floor attached. Raw containment without the floor is a vanity metric.
What good looks like: 60–80% containment with bot CSAT holding near human CSAT.
How to act on it: Always read containment and CSAT as a pair. If containment climbs while CSAT slides, you’re abandoning conversations, not resolving them. Roll back the last scope expansion.
8. First-response time
What it measures: Wait for the first reply — from the bot (near-instant) and, after escalation, from a human.
What good looks like: Bot under 2 seconds; human after handoff under 60 seconds.
How to act on it: Slow human pickup is a routing problem, but customers blame the bot. Fix queue routing or narrow the bot’s escalation hours to when humans are actually available.
9. Conversation abandonment rate
What it measures: Share of started conversations the visitor leaves before resolution — the chat equivalent of a bounce.
What good looks like: Under 20% for reactive chat.
How to act on it: Spikes trace to three causes: mistimed proactive triggers, the bot looping on misunderstood questions, or flows that ask too much. Check trigger timing first.
10. Goal completion rate
What it measures: Share of conversations ending in the designed outcome — a captured lead, a booked call, a purchase.
What good looks like: Above the same goal’s completion rate without chat.
How to act on it: Map drop-offs inside the flow like a checkout funnel. Most failures cluster where the bot asks for information before delivering value — flip the order: give first, ask second.
11. Repeat-contact rate (7-day)
What it measures: Customers returning about the same issue within seven days. The honesty check on every “resolution” metric.
What good looks like: Under 10% on automated conversations.
How to act on it: High repeat contact plus high “resolution” means your resolution definition is lying. Tighten it: no repeat contact, no escalation, no negative sentiment in the follow-up.
12. CSAT trend after training updates
What it measures: Direction of satisfaction after each retraining round. Training can make things worse.
What good looks like: Non-decreasing. Every iteration should hold or lift CSAT on the intents it touched.
How to act on it: Treat training like a release: change, measure, keep or revert.
The 5 vanity metrics to ignore
1. Total chat volume
More conversations isn’t better — a confusing checkout generates volume too. It measures activity, not value. Useful for capacity planning, not for success.
2. Raw deflection rate
The most gamed number in the industry. A customer who closes the chat in frustration counts as “deflected.” Without true-resolution and CSAT guardrails, it measures how good your bot is at ending conversations — not resolving them.
3. Number of intents
Four hundred intents sounds impressive until you learn half fire fewer than ten times a month. Intent count measures catalog size, not competence. Ten well-trained, high-volume intents beat a hundred thin ones.
4. Uptime percentage
99.9% uptime says the servers were on. It says nothing about answer quality, satisfaction, or revenue. Infrastructure hygiene — check it when something breaks, not in your weekly review.
5. Bot “intelligence” scores
Proprietary vendor scores with no external benchmark, designed to make renewals pleasant. Your CSAT split and repeat-contact rate are the only intelligence scores that matter.
Your weekly review routine (20 minutes)
- Quality (5 min): True resolution rate and the CSAT split. Top failing intents get retraining tickets.
- Money (5 min): Chat-engaged conversion and revenue per session vs. unassisted. Anything below 1x gets its triggers paused.
- Cost (5 min): Cost per resolved conversation and containment-with-CSAT. Containment up but CSAT down? Roll back the last scope change.
- Improvement (5 min): CSAT trend since the last training update. Flat or down? Retrain before the next cycle.
Oscar Chat tracks resolution, CSAT, and revenue per conversation out of the box — start free and check your numbers this week.
FAQ
How many chatbot metrics should a small team track?
Start with four: true resolution rate, the CSAT split, chat-engaged conversion rate, and cost per resolved conversation. They cover quality, revenue, and cost.
What’s a good chatbot resolution rate?
70–85% on routine intents like order status and returns. Above 90% deserves skepticism — verify “resolved” excludes frustrated abandonments.
How do I measure chatbot ROI?
Add both streams: support cost saved (lower cost per resolved conversation vs. your pre-bot baseline) plus incremental revenue from chat-engaged sessions. If either is flat after 90 days, your automation scope is usually wrong.
Should my bot’s CSAT match my human agents’?
Within about five points is the bar. A persistent double-digit gap means customers experience the bot as an obstacle — close it with retraining before expanding scope.
How often should I review chatbot analytics?
Weekly for the core four (the routine above); monthly for the full twelve, tied to your training cycle. Daily checks invite overreaction to noise.
Can a high containment rate be a bad sign?
Yes — when it rises while CSAT falls. That means the bot ends conversations instead of resolving them. Always read containment with the CSAT split, and roll back expansions that move them in opposite directions.
