Monitor and Fine-tune

Refine and validate AI Chatbot Q&A Sets for improved accuracy and quality, providing control and visibility into the value and effectiveness of the AI Chatbot.
The AI Chatbot "Monitor and Fine-tune" feature enables AI Admins to refine and validate AI Chatbot Q&A Sets for improved accuracy and quality, providing control and visibility into the value, user satisfaction, and overall effectiveness of the AI Chatbot.

Requirements:

SysAid Copilot license with AI Chatbot feature enabled

In this article, you’ll learn how to:

• Monitor AI Chatbot Q&A Sets

• Verify Q&A Set Relevance

Monitoring AI Chatbot answers

To open a Q&A Set’s complete content and metadata, click on any record in the table and the Q&A Set’s drawer will expand from the right-hand side.

This is where you’ll see the following:

Search and filter

You can search, filter, or sort the Monitor and Fine-tune table by the Q&A Set record’s Verification status:

• Unattended (default)

• Verified

• Irrelevant

• All Entries (no filter)

Filtering the table allows you to narrow down the displayed records to monitor the most relevant conversations.

You can also sort the table by column to view the table records according to your desired hierarchy.

Verifying query & answer (Q&A) sets

AI Admins can edit, verify, and submit AI Chatbot quarries and answers or mark them as Irrelevant.

User ranking and textual feedback

AI Admins can gauge user satisfaction with the AI Chatbot’s answer to each query when the end user:

• Clicks either “Thumbs Up” or “Thumbs Down” (that appears to the right of each AI Chatbot Answer)

• Provides textual feedback in the “Please share your feedback” lightbox that appears after clicking “Thumbs Up” or “Thumbs Down”

The ‘Thumb Up'/’Thumbs Down’ activity appears in:

• Verification Drawer for all Chat threads that a user ranked with ‘Thumbs Up’/’Thumbs Down’

• Monitor & Fine Tune table

• Expanded Quality Score view (appears upon hovering over Quality Score chip)

The textual feedback appears in a text bubble that appears when you hover over the Verification Drawer’s User Feedback chip.

End User Response Rating in AI Chatbot via Microsoft Teams:

AI Chatbot via Microsoft Teams Users have the option to rate the AI Chatbot’s answers directly from the Chat by clicking Thumbs Up or Thumbs Down.

This activity appears in the Monitor & Fine-Tune Table and in each Chat’s Verification Drawer.

Viewing answer sources

Each Q&A set includes two additional tabs, Sources and AI Agents, that offer transparency into how the chatbot generated its response.

These insights help you understand and evaluate the reasoning behind the answer, as well as the components that contributed to its creation.

Sources Tab

The Sources tab lists the different knowledge elements the chatbot considered when generating its response. These may include verified answers, knowledge base articles, templates, or other indexed content from your data pool.

Items actively used in the final answer are marked as “Used”, clarifying what shaped the chatbot’s response. The list of sources is influenced by the Secured Spots for Related References and Matching Precision settings configured in each database.

Tip:

To learn more about the Secured Spots for Related References and Matching Precision setting, see AI Advanced Configurations.

This setting defines a fixed number of references from a given dataset that should always be prioritized for consideration when forming a response. While these secured references are not guaranteed to appear in the final answer, they are reviewed with higher priority, helping ensure that critical information sources are consistently factored into the chatbot’s decision-making process.

By analyzing the Sources tab, you can fine-tune the quality and relevance of chatbot responses—whether by adjusting secured spots, improving content tagging, or updating verified answers.

AI Agents Tab

The AI Agents tab shows which AI Agents were triggered to fulfill the user request. This is especially useful when working with the AI Agent Builder, as it gives visibility into the AI Agent’s logic that contributed to the response. For example, if the user requested to reset their account password, the tab will list the name of the AI Agent(s) involved:

• Self-Service Azure ID Password Reset

• Self-Service - Asset Hardware Diagnostics

Tip:

To learn about AI Agents and the AI Agent Builder see:

• Prebuilt AI Agents Overview

• AI Agent Builder Guide

This tab is particularly useful for debugging or evaluating AI Agent performance. It allows you to confirm whether the correct AI Agent was invoked and whether it ran as expected. If the chatbot didn’t behave as intended, reviewing this tab can help pinpoint misconfigurations in the AI Agent logic.

AI Agents actively used in the final answer are marked as Used, clarifying what shaped the chatbot’s response. You will often see an (i) icon next to AI Agents that were executed by the LLM. Clicking on it opens the AI Agent execution details where the user can see the input parameter used for the agent’s execution and the output.

Modifying AI Chatbot queries & answers

AI Admins can modify (fine-tune) Q&A Sets while reviewing them. Modifying the Q&A Set before clicking “Submit Fine-tuned Answer” allows you to improve its content before it’s incorporated into the AI Chatbot Data Pool.

To modify an AI Chatbot Q&A, click over the content’s text area and change the desired text. You can also use the “AI Author” AI bubble.

Fine-tunning answers

Answers marked as Fine-tuned are submitted to the Data Pool so that it knows to prioritize the answer content over others for future user queries.

To verify an Answer, select Submit Verified Answer at the bottom of the conversation’s drawer. The Answer’s record will then be indicated as Verified in the Table and appear in the ‘Verified’ table view.

Marking as ‘Irrelevant’

AI Admins can mark AI Chatbot Q&A Sets as Irrelevant, which will: 

• Add it to the “Irrelevant” Table View

• Remove that Set from: 

  • AI Chatbot Data Pool

  • “Unattended” Table View

To mark an Answer as Irrelevant, select “Mark as Irrelevant” at the bottom of the Q&A Set’s drawer.

Please note:

Chatbot answers such as “Hi,” “How are you?” and “How can I help you?” are classified as pleasantries and automatically marked as “Irrelevant.” Since they are not an integral part of the exchange, they do not receive a quality score calculation.

To learn more about the AI quality score calculation, see Quality Score Calculation Overview.