No more crappy chatbots: metrics to keep in mind before your chatbot annoys customers again
Let’s be honest. Most chatbots suck. But not for the reason you think of.
It’s not always the result of poorly developed technology or because chatbots are not AI enough.
(Actually, I don’t understand the hype toward conversational AI when in reality, the thing that determines the positive chatbot experience is a good automation strategy.
There are many rule-based chatbots — the simplest type of chatbots today — that are winning people’s hearts. These chatbots just ask questions, and people answer them with buttons, and then the bot analyzes collected data and gives a reply.
Virgin Holidays used this type of chatbot for their marketing activity and won the Best Engagement Campaign 2018 by UK Social Media Communications Awards.)
My point is perfectly summed up in the famous quote by the Microsoft founder:
The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency.
That’s why the automation strategy is critical.
However, the second thing that determines the effectiveness and success of a chatbot is ongoing improvements.
Yes, you cannot just launch a chatbot on your website, app, or communicational channels and hope it will stick. No, even after a long testing period with focus groups, you still should keep an eye on how users, customers, patients — whoever — interact with your chatbot. And improve it constantly.
To know where and what to improve, you need to track and monitor chatbot analytics and the main chatbot metrics.
Here they are:
All these chatbot metrics are crucial.
Even more, some metrics are highly relying on others, so even if you see, for example, that many users chat with your chatbot BUT most of them are new ones, and the number of old users is low — your chatbot sucks.
Yes, you can complement your UX designer for making it appealing for people to start chatting. Still, you need to have a serious talk with your developers or conversational designers or account managers about why people refuse to use that chatbot to solve their problems.
Let’s dive into metrics.
Number of users
It’s a fundamental metric, but it gives you an understanding of how popular your chatbot is. Also, you can use this metric to calculate other metrics like conversion rate if you have a website chatbot. If this conversion rate is shallow, it may mean that chatbot looks unattractive, your welcome message looks unengaging, or simply users don’t notice your bot because of the website or widget design.
The more is better, but it’s not how we define chatbot success. You need your users to LOVE the chatbot. Or at least find it useful.
Usually, you can measure user satisfaction by doing surveys at the end of the conversation. You can also track user satisfaction during a conversation after some replies that a chatbot performs. However, don’t overload the chatbot with surveys and rating options. Just focus on the most critical times when you want the users to rate the chatbot.
Accuracy of a chatbot
This metric is relevant only if you have that *intelligent chatbot. I mean AI and NLP-powered chatbots.
Because these chatbots have to understand what users type, interpret their inputs correctly, and provide a response, you need to measure the responses’ accuracy.
Did the chatbot understand what the user was writing?
Did the chatbot provide the correct answer?
Accuracy is the MUST for AI chatbots in the customer service industry. If it cannot handle that, people will quickly figure out that using the chatbot for troubleshooting is meaningless. And they will start calling your overworked agents.
So, these are the most basic metrics you should track. Now, let’s focus on how to measure the engagement level of your chatbot.
If your chatbot is not one-day-thing, then you need to track how many people go back to your chatbot. This metric shows a count of unique users who send a message in a defined time frame. As well as with total users, you can track the active user’s number by itself or calculate the percentage of active users out of total users, which will give you the broader picture.
Average conversation duration
Conversation duration is the time from the user’s first message to the bot’s last message, and it should be calculated after the conversation ends. There is no benchmark for this metric because it’s very individual.
Don’t think that the longer the conversation duration with your chatbot, the better because it can mean that your chatbot is complex, and people need much time to find the answers they are looking for. Or sometimes people type stupid, awful things and are so lonely. They cannot resist talking to someone who responds 24/7.
The same logic with short conversation length — it doesn’t mean the chatbot underperforms. Maybe the users find the information they need in your chatbot fast, or you may have a short conversational flow.
So I wouldn’t make any major decisions based only on this metric.
It’s better to combine this metric with the metric below.
Goal Completion Rate (GCR)
This chatbot metric captures the percentage of successful engagement through a chatbot, meaning how many of your users reached their or your goals.
For example, your goal could be for a user to subscribe to your services, book a meeting with you, resolve a customer issue, etc. The GCR metric shows how successful your chatbot was in helping users reach these goals. If your GCR is low, don’t panic just yet. Try to find the reason why the chatbot isn’t reaching your goals.
Maybe the conversational flow is too confusing or lengthy. Perhaps the goal you set isn’t naturally expected or logical at that point. And keep in mind that people can have many goals when they go to chatbots. Just analyze your user chatbot journey, and you might find what’s stopping your users from reaching the goals you designed.
This metric measures the percentage of messages when the bot didn’t get user intent or failed to provide an answer to the user’s question. Or it simply didn’t have any information on the topic asked. You can use this metric to find areas for chatbot improvements and your business. Maybe people asking some questions that are not directed to the chatbot but to you?
Like, if people ask your chatbot about pricing, maybe you should create a page with pricing, hah?
For now, these are basic chatbot metrics you can start for further improvements. Sure, there are more, like Bounce Rate, Human Takeover Rate, ROI, and even more metrics depending on your case and industry.
Here are additional resources you can read to know more on the topic:
And, please, no more crappy chatbots!