Monitoring AI Chatbot Queries and Answers

Monitor & Fine-tune gives System Admins and AI Admins clear visibility into how the AI Chatbot is performing and where it needs attention. By surfacing underperforming answers, guardrail violations, and detailed response breakdowns, this interface makes it easy to review, improve, and continuously train the chatbot based on real user interactions.

Instead of guessing how the chatbot behaves in the wild, Monitor & Fine-tune turns every user query into actionable insight. You can identify weak answers, understand why they scored poorly, refine them, and feed improved responses back into the AI Chatbot Data Pool to raise answer quality over time.

How it works

System Admins and AI Admins use the Monitor & Fine-tune page to view, review, and improve real chatbot interactions. Each interaction between a user and the AI Chatbot is captured as a Q&A Set and made available here for monitoring, analysis, and refinement.

At the top of the page, you can control which chatbot interactions you want to review and how they are displayed. This allows each chatbot to be monitored and fine-tuned independently, based on its audience and use cases.

The page includes:

Tabs
The page includes multiple tabs that define the review scope:

• Flagged (open by default): Displays Q&A sets with low quality scores or other indicators that require attention.

• All: Displays all Q&A sets, including Verified, Flagged, and unreviewed items. Available to Admins and AI Admins only.

• Guardrail violations: Displays Q&A sets that violated guardrails defined for the selected chatbot, helping enforce compliance and safety rules.

Filters & search
You can filter Q&A sets by table columns, including date, query source, quality score, user feedback, and more. Filters help narrow the list to the interactions most relevant to your review.

The search bar lets you search for keywords across both user queries and chatbot responses, making it easy to locate specific topics, phrases, or issues.

Export
Click the export icon in the top-right corner of the page to export the currently filtered view as a CSV file for offline analysis, sharing, or reporting.

Reviewing Q&A sets flow

Once you’ve narrowed down the list, you can begin the review flow:

1. Identify a Q&A set for review
   Start with the Flagged tab to focus on underperforming answers that may need improvement.

2. Open the Q&A set details
   Click a row to open the chat information side panel, which provides a full breakdown of the interaction.

3. Analyze the interaction
   From the side panel, you can:

   • View and edit the user query and chatbot answer

   • Review the quality score and its contributing factors

   • See which sources and AI Agents were considered and used

   • View the full answer prompt for deeper context

4. Fine-tune and act
   After reviewing the interaction, you can refine the answer, add it to the Q&A dataset to improve future responses, or remove the item from the Flagged list once it has been addressed.

This flow allows you to continuously monitor chatbot performance, correct weak responses, and reinforce high-quality answers over time.

Answer sources

The Sources tab lists the different knowledge elements the chatbot considered when generating its response. These may include verified answers, knowledge base articles, templates, or other indexed content from your data pool.

Items actively used in the final answer are marked as “Used”, clarifying what shaped the chatbot’s response. The list of sources is influenced by the Secured Spots for Related References and Matching Precision settings configured in each database.

Tip!

To learn more about the Secured Spots for Related References and Matching Precision setting, see AI Advanced Configurations.

This setting defines a fixed number of references from a given dataset that should always be prioritized for consideration when forming a response. While these secured references are not guaranteed to appear in the final answer, they are reviewed with higher priority, helping ensure that critical information sources are consistently factored into the chatbot’s decision-making process.

By analyzing the Sources tab, you can fine-tune the quality and relevance of chatbot responses, whether by adjusting secured spots, improving content tagging, or updating verified answers.

AI agents

The AI Agents tab shows which AI Agents were triggered to fulfill the user request. This is especially useful when working with the AI Agent Builder, as it gives visibility into the AI Agent’s logic that contributed to the response. For example, if the user requested to reset their account password, the tab will list the name of the AI Agent(s) involved:

• Self-Service Azure ID Password Reset

• Self-Service - Asset Hardware Diagnostics

Tip!

To learn about AI Agents and the AI Agent Builder, see:

• Prebuilt AI Agents Overview

• AI Agent Builder Guide

This tab is particularly useful for debugging or evaluating AI Agent performance. It allows you to confirm whether the correct AI Agent was invoked and whether it ran as expected. If the chatbot didn’t behave as intended, reviewing this tab can help pinpoint misconfigurations in the AI Agent logic.

AI Agents actively used in the final answer are marked as Used, clarifying what shaped the chatbot’s response. You will often see an (i) icon next to AI Agents that were executed by the LLM. Clicking on it opens the AI Agent execution details, where the user can see the input parameter used for the agent’s execution and the output

Monitoring query and answer sets

By default, the Monitor & Fine-tune page opens on the Flagged tab. This tab acts as a focused “for review” list, showing Q&A sets with low quality scores or other issues that require attention. This ensures reviewers spend their time where it matters most, rather than scanning through high-performing answers.

To start fine-tuning query-answer sets:

1. Click the 3-dot menu in the top-right corner of the chatbot, then select Monitor & Fine-tune.

2. Using the drop-down in the top-left corner, choose which chatbot’s Q&A sets to review:

   • AI Chatbot for End Users

   • AI Chatbot for Agents

3. Click on an item in the table to open the Chat information side panel, where you can fully analyze the interaction and perform the following actions: