We are happy to announce that you can now evaluate the quality of your Xatkit bots using Botium. Thanks to Botium, you can easily create test cases for Xatkit, run your tests in the terminal or a web dedicated app, and integrate them into your CI/CD pipeline. Bots and chatbots are, in the end, software artefacts and, as such, they must be subjected to the same testing and Quality Assurance practices applied on other types of software applications.
Our Botium connector for Xatkit is available in NPM repository. Botium allows the definition of test cases using a textual DSL and provides a set of tools to manage and run such test cases. A test definition consists of a conversation simulation between a user and the bot. The example below shows a test case definition for the GreetingsBot.
Success conversation #me Hi #bot Hi, nice to meet you! #me How are you? #bot I am fine and you?
The conversation contains the user intents (preceded by #me) and the expected bot responses (preceded by #bot) displayed in the order they should occur. When running the test, Botium will connect to your chatbot and then bootstrap a conversation using the intent defined in the test definition. After each exchange, Botium will check the bot response against the expected reply defined in the test. A test succeeds when the conversation happens like it was described in the definition. The example above illustrates a simple conversation based on simple text only.
Our integration supports also the simulation of buttons and images (see a concrete example here). Find out more about how to write test cases in Botium here.
Xatkit tests can used in all Botium stack which includes:
- Botium CLI: a command line tool that provides many functionalities such running test suites directly in the terminal.
- Botium Bindings: a library that allows the binding of Botium tests with JS test runners (Mocha, Jasmine and Jest).
- Botium Box: a web based management and reporting app for Botium.
To start testing your chatbots with Xatkit and Botium, follow these steps:
-
- Install Botium CLI:
npm install -g botium-cli
- Install Xatkit connector:
npm install -g botium-connector-xatkit
- Create your Botium configuration file (i.e., botium.json).XATKIT_SERVER_URL should point to the URL of your Xatkit server.
{ "botium": { "Capabilities": { "PROJECTNAME": "botium-connector-xatkit-sample", "CONTAINERMODE": "xatkit", "XATKIT_SERVER_URL": "http://localhost:5001" } } }
That’s it! you’re good to go. You can now use Botium CLI to chat with your bot and save the conversation as a test case using botium-cli emulator command:
You can also run your test cases using botium-cli run command:
Refer to the official documentation of Botium to know more about how to test chatbots using the Botium Stack. A sample project that uses Botium-bindings with Xatkit and Mocha test runner can be found here.
With this, you’ll have the core infrastructure to create and run tests for your bots and make sure they achieve the required level of bot quality. And as important as that, to make sure they maintain that level of quality when releasing new versions of the bot as sometimes new intents could interfere with existing ones, e.g. because their training sentences are too similar (kind of regression testing for chatbots). Until this is fully automated, you could still keep an eye on the bot monitoring data in our dashboard.
