Monitor and Fine-tune
- 18 Dec 2024
- Print
- PDF
Monitor and Fine-tune
- Updated on 18 Dec 2024
- Print
- PDF
Article summary
Did you find this summary helpful?
Thank you for your feedback
Refine and validate AI Chatbot Q&A Sets for improved accuracy and quality, providing control and visibility into the value and effectiveness of the AI Chatbot.
The AI Chatbot "Monitor and Fine-tune" feature enables AI Admins to refine and validate AI Chatbot Q&A Sets for improved accuracy and quality, providing control and visibility into the value, user satisfaction, and overall effectiveness of the AI Chatbot.
Requirements:
SysAid Copilot license with AI Chatbot feature enabled
In this article, you’ll learn how to:
Monitor AI Chatbot Q&A Sets
Verify Q&A Set Relevance
Access Monitor & Fine-tune
To access the Monitor & Fine-tune area:
In SysAid Spaces
Go to Settings > SysAid Copilot > Monitor & Fine-tune.
In SysAid Classic
Go to Settings > Service Desk > General.
Scroll all the way down and click Configure SysAid Copilot Settings.
Click the gear icon in the bottom left corner.
Click Monitor & Fine Tune
Monitor AI Chatbot answers
The Monitor and Fine-tune Table displays a list of all previous AI Chatbot Q&A Sets (both Query and Answer) as well as the AI Chatbot metadata (Quality Score and Verification).
To open a Q&A Set’s complete content and metadata, click on any record in the table and the Q&A Set’s drawer will expand from the right-hand side.
This is where you’ll see the following:
Item | Definition | Action |
Verification | SysAdmin can verify whether a Q&A Set is sufficient/relevant to the user query, and should be prioritized for recurring use in future queries | Select “Submit Fine-tuned Answer” or “Clear from List” |
Quality Score | Answer’s ‘score’; measured in percentage on a scale based on the following parameters:
| Hover over the Quality Score chip to view the individual parameter rankings |
Q&A Set | The Query and Answer texts in the AI Chatbot Conversation | Click on the text to edit |
User Ranking | Whether the user clicked “Thumbs Up” or “Thumbs Down” and provided AI Chatbot feedback | Hover over the chip in the Verification Drawer to read what the user typed in the Submit feedback field |
Image | SysAdmins can preview any image added to the chat by the end user. | Click the image icon to preview in a new tab |
Please note:
The “Unattended” label in the Query Drawer indicates that the Query has not been marked as “Verified” or as “Irrelevant”
The “AI Editor” AI bubble appears when the AI Admin clicks on the User Query or Answer texts
Chats containing an image include a textual description of the image for the SysAdmin to review and/or fine-tune.
Search, filter, or sort the Table
You can search, filter, or sort the Monitor and Fine-tune table by the Q&A Set record’s Verification status:
Unattended (default)
Verified
Irrelevant
All Entries (no filter)
Filtering the table allows you to narrow down the displayed records to monitor the most relevant conversations.
You can also sort the table by column to view the table records according to your desired hierarchy.
Verify Q&A Set Relevance
AI Admins can edit, verify, and submit AI Chatbot Answers or, alternatively, mark them as Irrelevant.
Verified Answers:
Verified answers can be modified and re-verified
Read User Ranking & Textual Feedback
.jpg)
AI Admins can gauge User Satisfaction with the AI Chatbot’s Answer to each Question when the End User:
Clicks either “Thumbs Up” or “Thumbs Down” (that appears to the right of each AI Chatbot Answer)
Provides textual feedback in the “Please share your feedback” lightbox that appears after clicking “Thumbs Up” or “Thumbs Down”
The ‘Thumb Up'/’Thumbs Down’ activity appears in:
Verification Drawer for all Chat threads that a user ranked with ‘Thumbs Up’/’Thumbs Down’
Monitor & Fine Tune table
Expanded Quality Score view (appears upon hovering over Quality Score chip)
The textual feedback appears in a text bubble that appears when you hover over the Verification Drawer’s User Feedback chip.
End User Response Rating in AI Chatbot via Microsoft Teams
AI Chatbot via Microsoft Teams Users have the option to rate the AI Chatbot’s answers directly from the Chat by clicking Thumbs Up or Thumbs Down.
This activity appears in the Monitor & Fine-Tune Table and in each Chat’s Verification Drawer.
Modify AI Chatbot Queries & Answers
AI Admins can modify (fine-tune) Q&A Sets while reviewing them. Modifying the Q&A Set before clicking “Submit Fine-tuned Answer” allows you to improve its content before it’s incorporated into the AI Chatbot Data Pool.
To modify an AI Chatbot Q&A, click over the content’s text area and change the desired text. You can also use the “AI Author” AI bubble.
Fine-tuned Answers
Answers marked as Fine-tuned are submitted to the Data Pool so that it knows to prioritize the answer content over others for future user queries.
To verify an Answer, select Submit Verified Answer at the bottom of the conversation’s drawer. The Answer’s record will then be indicated as Verified in the Table and appear in the ‘Verified’ table view.
Mark as Irrelevant
AI Admins can mark AI Chatbot Q&A Sets as Irrelevant, which will:
Add it to the “Irrelevant” Table View
Remove that Set from:
AI Chatbot Data Pool
“Unattended” Table View
To mark an Answer as Irrelevant, select “Mark as Irrelevant” at the bottom of the Q&A Set’s drawer.
Please note:
Chatbot answers such as “Hi,” “How are you?” and “How can I help you?” are classified as pleasantries and automatically marked as “Irrelevant.” Since they are not an integral part of the exchange, they do not receive a quality score calculation.
To learn more about the AI quality score calculation, see Quality Score Calculation Overview.
.png)