Measuring Bot Health
While using a simplified health score to measure chatbot success is convenient, it's essential to understand the critical signals that influence this score. These signals help us identify, prioritize, and address issues within the bot, ultimately improving self-service rates. Higher self-service rates lead to greater impact and cost savings.
Understanding Critical Signals
Critical Signals and their definitions
- Completion Rate: Measures how often users complete their conversations with the bot.
- Sentiment Value: Assesses user emotions during interactions, indicating whether they are expressing positive or negative feelings.
- User Sentiment: Measures the overall emotional tone of user interactions, indicating satisfaction or frustration. This insight helps improve user experience by identifying areas that need enhancement to increase positive sentiment.
- Helpfulness: An AI-enriched metric that evaluates whether the bot's responses are genuinely helpful in addressing user queries.
- Effort: Tracks the efficiency of the resolution process, measuring how long or how many interaction turns it takes to resolve a user's issue.
Completion of User's Requests | User Sentiment | Helpfulness | Effort |
---|---|---|---|
Full Completion: Whether all the user's requests were fully met. | Positive Indicators: Identifying expressions of satisfaction, gratitude, or relief. | Accuracy: Checking if the provided information was correct and pertinent to the user's requests. | Response Time: Measuring the speed at which the user's requests were addressed. |
Partial Completion: Identifying if only some aspects of the user's requests were addressed. | Negative Indicators: Noting any complaints, frustration, or dissatisfaction. | Clarity: Evaluating how clearly the information was communicated. | Interaction Efficiency: Assessing the number of exchanges needed to resolve the requests. |
AI Leveraged Signals
These critical signals are enhanced by our AI models, providing insights into the following areas:
- Health Score: A comprehensive indicator of overall bot performance.
- Top Satisfaction Drivers: Key factors contributing to positive user experiences.
- Top Dissatisfaction Drivers: Common reasons behind user frustration or dissatisfaction.
Tracking Areas for Improvement
- Transfer Rate: calculates the percentage of conversations where users are escalated to live agent support.
- Transfer Rate Drivers: Identifies categories and reasons that cause users to transfer to agents, highlighting the biggest issues needing attention.
Tracking What's Working
- Self-Service Rate: Measures how many conversations are successfully handled by the bot without human intervention, including early drop-offs.
- Top Self-Service Drivers: Highlights the top use cases that lead to successful self-service interactions.
Conclusion
The health score can increase or decrease based on these critical signals. By monitoring them through the bot health dashboard, you can effectively track and improve your bot's performance.
Updated 29 days ago
What’s Next