- 21 Feb 2024
Monitor and Fine-tune
- Updated on 21 Feb 2024
Refine and validate AI Chatbot Q&A Sets for improved accuracy and quality, providing control and visibility into the value and effectiveness of the AI Chatbot.
The AI Chatbot "Monitor and Fine-tune" feature enables AI Admins to refine and validate AI Chatbot Q&A Sets for improved accuracy and quality, providing control and visibility into the value, user satisfaction, and overall effectiveness of the AI Chatbot.
SysAid Copilot license with AI Chatbot feature enabled
This document describes how to:
Access Monitor & Fine Tune
To navigate to the Monitor and Fine Tune Table, click the Settings (cogwheel) icon on the left-hand sidebar of the AI Chatbot screen, and click Monitor & Fine Tune.
Monitor AI Chatbot Q&A Sets
The Monitor and Fine-tune Table displays a list of all previous AI Chatbot Q&A Sets (both Query and Answer) as well as the AI Chatbot metadata (Quality Score and Verification).
To open a Q&A Set’s complete content and metadata, click on any record in the table and the Q&A Set’s drawer will expand from the right-hand side.
This is where you’ll see the following:
AI Admins can verify whether a Q&A Set is sufficient/relevant to the user query, and should be prioritized for recurring use in future queries
Mark as “Verified” or “Irrelevant”
Answer’s ‘score’; measured in percentage on a scale, based on the following parameters:
Hover over Quality Score chip to view the individual parameter rankings
The Query and Answer texts in the AI Chatbot Conversation
Click on text to edit
Whether the user clicked “Thumbs Up” or “Thumbs Down” to provide AI Chatbot feedback
Hover over chip to read what user typed in the “Submit feedback” input field
The “Unattended” label in the Query Drawer indicates that the Query has not been marked as “Verified” or as “Irrelevant”
The “AI Editor” AI bubble appears when the AI Admin clicks on the User Query or Answer texts
Search, filter or sort the Table
You can search, filter, or sort the Monitor and Fine-tune table by the Q&A Set record’s Verification status:
All Entries (no filter)
Filtering the table allows you to narrow down the displayed records so that you can monitor the most relevant conversations.
You can also sort the table by Column, to view the table records according to your desired hierarchy.
Verify Q&A Set Relevance
SysAdmins can edit, verify and submit AI Chatbot Answers, or alternatively, mark it as Irrelevant.
Verified answers can be modified and re-verified
Read User Ranking & Feedback
AI Admins can gauge User Satisfaction with the AI Chatbot’s Answer to each Question when the End User:
Clicks either “Thumbs Up” or “Thumbs Down” (that appears to the right of each AI Chatbot Answer)
Provides textual feedback in the “Please share your feedback” lightbox that appears after clicking “Thumbs Up” or “Thumbs Down”
These two forms of feedback are calculated into the Q&A Set’s Quality Score, and appear in the Verification Drawer for all Chat threads that received User Feedback from the End User.
To read the End User’s textual feedback, hover over the User Feedback chip in the Verification Drawer and the feedback text will appear in a small text bubble.
Modify AI Chatbot Queries & Answers
AI Admins can modify (fine-tune) Q&A Sets while reviewing them. Modifying the Q&A Set before clicking “Submit Verified Answer” allows you to improve its content before it’s incorporated in the AI Chatbot Data Pool.
To modify an AI Chatbot Q&A, click over the content’s text area and change the desired text, with the option of using the “AI Author” AI bubble.
Answers marked as “Verified” are submitted to the Data Pool so that it knows to prioritize the Answer content over others for future user queries.
To verify an Answer, select “Submit Verified Answer” at the bottom of the conversation’s drawer. The Answer’s record will then be indicated as Verified in the Table and appear in the ‘Verified’ table view.
Mark as Irrelevant
AI Admins can mark AI Chatbot Q&A Sets as Irrelevant, which will:
Add it to the “Irrelevant” Table View
Remove that Set from:
AI Chatbot Data Pool
“Unattended” Table View
To mark an Answer as Irrelevant, select “Mark as Irrelevant” at the bottom of the Q&A Set’s drawer.