Most chatbot platforms focus, understandably, on all the cool features they offer to create great chatbots. But IMHO, a chatbot without proper monitoring is worth next to nothing.
When you start creating a chatbot you think
- You know what your visitors will want to ask your bot and
- How will they express those questions
You are wrong and wrong. Visitors will always surprise you. So, creating a chatbot and deploying it in the wild without supervision is worthless (IMHO).
During the first iterations, a chatbot should be seen as a discovery tool to grasp what the users really want to know. Once this is done, you can create the actual chatbot that will reply those questions
Why do you need to monitor your chatbot?
Every chatbot project should come with a clear monitoring plan to identify frequent utterances that the chatbot is unable to match to see whether those utterances are:
- Out of scope questions. Nothing important to do here except printing some kind of error message clarifying the focus of the bot
- Wrong matches. The user input is matched against the wrong intent giving him an answer that it’s confusing or just plain worng. This situation is very important (but difficult) to detect without a proper monitoring tool.
- Interesting new questions that are not part of the current set of chatbot intents. The bot should be extended to include a proper answer to the question
- An utterance that the bot should have indeed matched as it corresponds to one of the intents already included in the bot. The training set for that intent should be expanded to cover the alternative wording employed by the visitor.
To detect all these situations, the chatbot platform must offer some kind of monitoring capabilities that the chatbot designer can exploit to put in place some kind of continuous improvement process for the chatbot. For instance, Xatkit comes with a monitoring REST API you can use to get all kinds of analytics data, session data, unmatched utterances and various stats about the bot behavior.
This is enough to build your own chatbot monitoring dashboard. But as this requires some technical skills, the clients of our managed chatbot hosting plans have access to a dedicated graphical chatbot dashboard that facilitates the quick identification of the key areas of improvement for the chatbot.
A graphical dashboard for your chatbots
Our dashboard comprises several chatbot analysis sections, each one highlighting a different aspect of the bot behavior. The dashboard is fully configurable so we can easily adapt it to your specific needs if so desired. Let’s see some of these default data sections computed and rendered by the dashboard.
Global data
Overview of bot usage and overall behavior. We can see the distribution of utterances between matched and unmatched (hovering over the corresponding sections gives additional information like the exact number of messages of each type and the average confidence of the matched ones).
We also show the number of sessions opened in the bot and the average number of exchanges in each session total and in a heatmap-like chart the changes in the number of bot users in the last seven days.
Intent Data
This section helps you understand what are the intents most typically target by the chatbot visitors. In the case of complex chatbots, you can filter out hardly used intents to get a better picture.
Besides, when clicking in the pie, you can get specific utterances information about a given intent, including the timestamp, the matched utterance and the confidence level of the match.
A second chart in this section helps you see how the intents relate to each other, based on actual usage. This is especially useful for “open” bots where there is no strict conversation path and we want to understand how users freely navigate within the bot.
Clicking in one of the intents zooms in in its relationships and clarifies what are the common previous and next intents to the selected one.
Utterances Data
This part of the dashboards reports on the utterances entered by the chatbot users. The core of this section shows a table with all the utterances from last days, showing fore each utterance: its timestamp, the actual user text, the matching intent and the confidence level.
User Behavioural Data
This section focuses on data analyses that help you understand when the users quit the bot so that the chatbot makes a better job of keeping them engaged next time. For instance, as you see in this graphic, the chatbot monitoring dashboard shows the last step reached during the conversation before the user disappeared, how many times the user tried to match the next intent and how long before he gave up definitely.
You also have the option to dig deeper into each one of the sessions that ended up with the user quitting the chatbot. You can see exactly what was the conversation path he followed and what utterance he entered at every single step of the way.
Trends Data
Finally, the dashboard shows some stats to help you evaluate whether the chatbot is getting more or less popular and whether it is performing better and better over time (in terms of the number of sessions, successful matches, confidence level,…).
And as for the chatbot itself, you can see all this data with the dark mode on or switch to a more standard white background chatbot dashboard.
