{"id":120109,"date":"2021-12-20T16:02:48","date_gmt":"2021-12-20T16:02:48","guid":{"rendered":"https:\/\/xatkit.com\/?p=120109"},"modified":"2021-12-29T15:44:07","modified_gmt":"2021-12-29T15:44:07","slug":"chatbot-open-question-qa-haystack","status":"publish","type":"post","link":"https:\/\/xatkit.com\/chatbot-open-question-qa-haystack\/","title":{"rendered":"How to build a chatbot that reads all your data to find the right answer"},"content":{"rendered":"<p>I&#8217;m sure you often get questions from your visitors and think &#8220;but this is already on the website!&#8221;. You then added a chatbot to your site to filter out the <a href=\"https:\/\/xatkit.com\/pareto-principle-chatbot-intent-design\/\" target=\"_blank\" rel=\"noopener\">most common questions<\/a>. But what about all the rest. Adding more and more questions to the bot it takes time. So, is there an easy way to <strong>let the chatbot find the answer by itself<\/strong> for those questions that are already answered somewhere in the (hundreds? thousands?) pages, posts or other online documents you&#8217;ve already published?<\/p>\n<p>YES!. The key is to plug <a href=\"https:\/\/haystack.deepset.ai\/\" target=\"_blank\" rel=\"noopener\" data-schema-attribute=\"\">Haystack<\/a> to your chatbot. We have several <a href=\"https:\/\/huggingface.co\/models?pipeline_tag=question-answering\" target=\"_blank\" rel=\"noopener\">pretrained language models fine-tuned for Question Answering<\/a> that can be used to find an answer in a text but they only work when the text length is really small. Here it is where Haystack comes into play. Haystack architecture (see the featured image above) proposes a two-phase process:<\/p>\n<ul>\n<li>A <em>Retriever<\/em> selects a set of candidate documents from all the available information. Among other options, we can rely on\u00a0<a href=\"https:\/\/www.elastic.co\/\" target=\"_blank\" rel=\"noopener\">ElasticSearch<\/a> to index the documents and return those that most likely contain the answer to the question<\/li>\n<li>A <em>Reader <\/em>applies state-of-the-art QA models to try to infer an answer from each candidate<\/li>\n<\/ul>\n<p>Now we just need to return these inferred solutions together with additional information (context, confidence level,&#8230;) to help our users understand why we think this is the answer they were looking for. All thanks to a completely <a href=\"https:\/\/github.com\/deepset-ai\/haystack\" target=\"_blank\" rel=\"noopener\">free and open source NLP framework!<\/a><\/p>\n<p>Let&#8217;s see how we can benefit from Haystack by creating a chatbot able to answer software design questions for the <a href=\"https:\/\/wordpress.org\/\" target=\"_blank\" rel=\"noopener\">WordPress<\/a> website <a href=\"https:\/\/modeling-languages.com\/\" target=\"_blank\" rel=\"noopener\" data-schema-attribute=\"\">modeling-languages.com<\/a>. Haystack website is full of useful examples, so we&#8217;ll adapt them in our scenario.<\/p>\n<h2>Creating the chatbot, the &#8220;front-end&#8221;<\/h2>\n<p>The easiest part is to create the chatbot. We&#8217;ll obviously use <a href=\"https:\/\/github.com\/xatkit-bot-platform\/xatkit\" target=\"_blank\" rel=\"noopener\" data-schema-attribute=\"\">Xatkit<\/a> for this. The bot can have as many intents as you wish. The only part that we care about here is the <a href=\"https:\/\/xatkit.com\/define-meaningful-fallbacks-for-your-chatbot\/\" target=\"_blank\" rel=\"noopener\" data-schema-attribute=\"\"><em>default fallback\u00a0<\/em><\/a>state. Here, instead of saying something useless, e.g. &#8220;sorry I didn&#8217;t get your question, can you rephrase it and try again?&#8221;, we will ask Haystack to find us a solution.<\/p>\n<p><script src=\"https:\/\/gist.github.com\/jcabot\/5d3ef3282e6430acf5a709755b188f49.js\"><\/script><\/p>\n<p>Here we have all the pieces locally deployed but obviously each of them could be in a different server.<\/p>\n<h2>Loading the information to ElasticSearch<\/h2>\n<p>Before we can find an answer, we need to first power up ElasticSearch with the documents we want to use as information source for the bot. In this example, the documents will be all posts published in modeling-languages. We assume we have direct access to the WordPress database but otherwise we could write something similar using the <a href=\"https:\/\/developer.wordpress.org\/rest-api\/\" target=\"_blank\" rel=\"noopener\" data-schema-attribute=\"\">WordPress REST API\u00a0<\/a>instead. Note that we split each post in different paragraphs to avoid chunks of text that could be too much for the QA models.<\/p>\n<p><script src=\"https:\/\/gist.github.com\/jcabot\/7ca0c3d437659eb27ca1b6e8fa047bbe.js\"><\/script><\/p>\n<p>I deployed a <a href=\"https:\/\/flask.palletsprojects.com\/en\/2.0.x\/\" target=\"_blank\" rel=\"noopener\" data-schema-attribute=\"\">Flask<\/a> server to facilitate calling all the endpoints on-demand, especially needed for those that the bot needs to interact with.<\/p>\n<h2>Finding the answer<\/h2>\n<p>The <em>Retriever<\/em> component will look for the most promising documents in ElasticSearch (by default, using the <a href=\"https:\/\/www.elastic.co\/blog\/practical-bm25-part-2-the-bm25-algorithm-and-its-variables\" target=\"_blank\" rel=\"noopener\" data-schema-attribute=\"\">BM25 algorithm<\/a> but there are other options). The <em>Reader<\/em> will look into each candidate and try to find the right answer in it. Thanks to the predefined <a href=\"https:\/\/haystack.deepset.ai\/reference\/pipelines\" target=\"_blank\" rel=\"noopener\" data-schema-attribute=\"\">Pipelines<\/a> provided by Haystack, putting everything together is really easy:<\/p>\n<p><script src=\"https:\/\/gist.github.com\/jcabot\/382c617c2cb753e47f23803d05c278e3.js\"><\/script><\/p>\n<p>Once we have the answer, we create the response object that will be sent back to the chatbot. As a final step the chatbot will print this response to the user together with the URL of the post the answer comes from. This way, even if the answer is not perfect the user will have the option to go to the suggested URL.<\/p>\n<h2>But, does it work?<\/h2>\n<p>We&#8217;ve seen it&#8217;s feasible to add the Haystack infrastructure to a chatbot. But what about the quality of the answers? Are they good enough?<\/p>\n<p>The answer is that it does work reasonably well. The modeling-languages website was not the easiest one to try with. It&#8217;s rather large (over 1000 posts that translate into around 8000 documents) with significant overlappings. And there are still some &#8220;legay&#8221; posts in Spanish that add to the confusion.<\/p>\n<p>Let&#8217;s see a couple of examples. In the first one I ask on how can I add a business rule (i.e. constraint) to a software design model. The first answer is technically correct (indeed, constraints are written on top of models) but rather useless. The next two are exactly what I was hoping to see an answer as they suggest me to use the Object Constraint Language to specify my constraints.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-120111 aligncenter\" src=\"https:\/\/xatkit.com\/wp-content\/uploads\/2021\/12\/OCLExample.png\" alt=\"Question Answering example on a WordPress site\" width=\"375\" height=\"632\" srcset=\"https:\/\/xatkit.com\/wp-content\/uploads\/2021\/12\/OCLExample.png 375w, https:\/\/xatkit.com\/wp-content\/uploads\/2021\/12\/OCLExample-178x300.png 178w\" sizes=\"(max-width: 375px) 100vw, 375px\" \/><\/p>\n<p>The second question is more concrete but it has a more open answer. Note that all answers are taken from the same document (the only one that realy talks about this Temporal EMF tool). All answers are reasonable but the third one really nails it. And keep in mind that we&#8217;re using an extractive QA model, meaning that the model aims to return a subset of the text containing the answer. Instead, a generative QA model (also available in Haystack) would be able to &#8220;build&#8221; the answer from partial answers, potentially spread out in more than document.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-120112 aligncenter\" src=\"https:\/\/xatkit.com\/wp-content\/uploads\/2021\/12\/temporalExample.png\" alt=\"QA haystack example with a more concrete question\" width=\"374\" height=\"625\" srcset=\"https:\/\/xatkit.com\/wp-content\/uploads\/2021\/12\/temporalExample.png 374w, https:\/\/xatkit.com\/wp-content\/uploads\/2021\/12\/temporalExample-180x300.png 180w\" sizes=\"(max-width: 374px) 100vw, 374px\" \/><\/p>\n<p>In terms of <strong>performance, results were very satisfactory<\/strong>. The whole process took just a few seconds (once the initial model loading) but this is on my poor laptop. With proper configuration and tuning, the user should not notice a major delay. <a href=\"https:\/\/haystack.deepset.ai\/guides\/rest-api\" target=\"_blank\" rel=\"noopener\">Haystack itself can also be deployed as a REST API<\/a> which should optimize even more the whole process.\u00a0 And of course, you could always let the bot designer to configure whether to use Haystack in the default fallback or not, depending on a number of factors.<\/p>\n<p>And remember that if you&#8217;re interested mianly in returning an orderet set of relevant documents for the user to check out (instead of providing the &#8220;exact&#8221; answer), the <a href=\"https:\/\/haystack.deepset.ai\/docs\/latest\/rankermd\" target=\"_blank\" rel=\"noopener\" data-schema-attribute=\"\"><em>Ranker<\/em> <\/a>is your friend. As the <em>Reader,<\/em> the <em>Ranker<\/em> executes after the <em>Retriever<\/em> and also uses pretrained language models but in this case with the goal of doing semantic analysis of the documents for sorting purposes.<\/p>\n<span class=\"et_bloom_bottom_trigger\"><\/span>","protected":false},"excerpt":{"rendered":"<p>Use Haystack to combine the latest pretrained language models with ElasticSearch to create bots able to answer almost any question.<\/p>\n","protected":false},"author":8,"featured_media":120110,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":""},"categories":[9,6,306],"tags":[344,346],"_links":{"self":[{"href":"https:\/\/xatkit.com\/wp-json\/wp\/v2\/posts\/120109"}],"collection":[{"href":"https:\/\/xatkit.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/xatkit.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/xatkit.com\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/xatkit.com\/wp-json\/wp\/v2\/comments?post=120109"}],"version-history":[{"count":0,"href":"https:\/\/xatkit.com\/wp-json\/wp\/v2\/posts\/120109\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/xatkit.com\/wp-json\/wp\/v2\/media\/120110"}],"wp:attachment":[{"href":"https:\/\/xatkit.com\/wp-json\/wp\/v2\/media?parent=120109"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/xatkit.com\/wp-json\/wp\/v2\/categories?post=120109"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/xatkit.com\/wp-json\/wp\/v2\/tags?post=120109"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}