1. Identify the website, data source or knowledge base which the Chatbot will refer to for answering questions.
2. Extract the data from the repository as a one-time export or in a continuous manner if the data is updated frequently.
3. Identify a suitable generative AI large language model (LLM) that can be fine-tuned on the data extracted from the knowledge repository. Confirm that the LLM you are selecting includes RAG capabilities (Retrieval Augmented Generation) so that the model’s responses reflect new information added or changed in your data repository.
4. Ensure that the data being used for fine tuning is secured from any personally identifiable information or protected data. It is usually best to do this by including a personally identifiable information (PII) masking model as part of the data repository ingestion process.
5. To enhance the LLM’s ability to respond empathetically to contextual and sentiment queues from the user, explore entity identification as well as sentiment and emotion analysis are being considered by the LLM’s generative responses and implement supplemental models if needed.
6. Consider your user experience and understand whether the chatbot will operate only with publicly available information (Do you offer petite sized polo shirts?) versus scenarios where the chatbot will need to support user-specific information (What size did I order when I last purchased a polo shirt?).  This is also true if your business has different audiences with very specific questions.  Consider creating different AI chatbots that draw from different source information.
   • For public sources, ensure the data extracted is tagged to the right source so the question addressed is not informed by information not intended for the public. This is critical to managing the “hallucinations” that occur when a response may include odd information seemingly unrelated to the general question but was informed by dated or specialized content.
   • For private use cases, ensure that the chatbot does not retrieve information from Account A when the question pertains to Account.

7. Once the fine-tuned model is ready, test for accuracy. Keep in mind that odd or inaccurate responses can usually be attributed to the source data containing contradictory or dated references. This AI chatbot example resulted in litigation and was probably related to the airline not having clearly identified refund policies to draw from and could have been easily managed.
8. As you test your model, you will want to fine-tune how it is responding. Design the Q&A experience using the best practices of prompt engineering to ensure the model responds with contextual and accurate information. For instance, if information is not found, make sure you provide the chatbot with a standard response that offers the user another path to getting that question answered accurately.
9. Ensure that the Q&A experience can maintain session integrity and retain information so that a conversational nature is established between the chatbot and the user. This ensures that the user does not have to repeat their question content when asking for additional information.
10. Connect the chatbot to the customer interface to receive submitted queries and display the responses.
11. Monitor the chatbot queries and responses for accuracy. If the LLM model you are using provides predictive accuracy you can use this to set informed monitoring thresholds.
12. Analyze for Sentiment, Emotion, Categorization, and Tone in the interactions between the user and the chatbot (constantly assess the chatbot for negative sentiments or emotions, and apply any compliance-related checks i.e. nothing that could be interpreted as healthcare or financial advice provided).
13. If the Sentiment & Emotion Analysis or the accuracy of the responses are not as expected, escalate the user’s query to a live agent to quickly help.
14. Improve the chatbot responses based on the analysis by extending the data sources and fine-tuning the prompts designed for the model.

If you would like to take advantage of an AI chatbot that is already configured and ready to be connected to your information and use case experience Qualetics can help.  

Qualetics will provision a fully developed AI Chatbot Data Machine and the Web based Chatbot User Interface. We will work with your team to connect our AI Chatbot Data Machine to the website or knowledgebase(s) of your choice, test and fine tune the experience, then help you integrate your AI chatbot with your application of your choice.