Monitor and Fine-tune
    • 01 Jan 2024
    • PDF

    Monitor and Fine-tune

    • PDF

    Article Summary

    AI Chatbot "Monitor and Fine-tune" feature

    The AI Chatbot "Monitor and Fine-tune" feature enables SysAdmins to refine and validate AI Chatbot Q&A Sets for improved accuracy and quality, providing the sysadmin with control and visibility into the value and effectiveness of the AI Chatbot.

    SysAdmins can access the Monitor and Fine-tune Table by clicking the cogwheel icon from the AI Chatbot interface.
    This document describes how to:

    • Monitor AI Chatbot Q&A Sets
    • Verify Q&A Set Relevance

    Access the Monitor and fine-tune table from the AI Chatbot screen To navigate to the Monitor and fine-tune Table, click the Settings (cogwheel) icon on the left-hand sidebar of the AI Chatbot screen, and click “Monitor and fine-tune”.

    Monitor AI Chatbot answers

    Monitor and Fine-tune Table

    The Monitor and Fine-tune Table displays a list of all previous AI Chatbot Q&A Sets (both Question and Answer) as well as the AI Chatbot metadata (Quality Score and Verification).

    To open a Q&A Set’s complete content and metadata, click on any record in the table and the Q&A Set’s drawer will expand from the right-hand side.

    This is where you’ll see the following:

    VerificationSysAdmin can verify whether a Q&A Set is sufficient/relevant to the user query, and should be prioritized for recurring use in future queriesMark as “Verified” or “Irrelevant”
    Quality Score

    Answer’s ‘score’; measured in percentage on a scale, based on the following parameters:

    • How much context was covered from the AI Chatbot’s Data Pool
    • Was the answer self-servable (user doesn’t require further assistance)?
    • Did the user need to submit a Service Record after the conversation?
    • End user ranking (thumbs up/thumbs down)
    Hover over Quality Score chip to view the individual parameter rankings
    Q&A SetThe Query and Answer texts in the AI Chatbot Conversation

    Click on text to edit

    • The “Unattended” label in the Query Drawer indicates that the Query has not been marked as “Verified” or as “Irrelevant”
    • The “AI Editor” AI bubble appears when the SysAdmin clicks on the User Query or Answer texts

    Search, filter or sort the Table 

    • Filter the Table by its Verification status

      You can search, filter, or sort the Monitor and Fine-tune table by the Q&A Set record’s Verification status:
      •  Unattended (default)
      • Verified
      • Irrelevant
      • All Entries (no filter)

      Filtering the table allows you to narrow down the displayed records so that you can monitor the most relevant conversations.

      You can also sort the table by Column, to view the table records according to your desired hierarchy.

      Verify Q&A Set Relevance

      Verify the relevance of a Q&A Set

      SysAdmins can edit, verify and submit AI Chatbot Answers, or alternatively, mark it as Irrelevant.

      Verified answers can be modified and re-verified

      Modify AI Chatbot Queries & Answers

      SysAdmins can modify (fine-tune) Q&A Sets while reviewing them. Modifying the Q&A Set before clicking “Submit Verified Answer” allows you to improve its content before it’s incorporated in the AI Chatbot Data Pool.
      To modify an AI Chatbot Q&A, click over the content’s text area and change the desired text, with the option of using the “AI Author” AI bubble.

      Verified Answers

      Answers marked as “Verified” are submitted to the Data Pool so that it knows to prioritize the Answer content over others for future user queries.

      To verify an Answer, select “Submit Verified Answer” at the bottom of the conversation’s drawer. The Answer’s record will then be indicated as Verified in the Table and appear in the ‘Verified’ table view.

      Irrelevant Answers

      SysAdmins can mark AI Chatbot Q&A Sets as Irrelevant, which will: 

      • Add it to the “Irrelevant” Table View
      • Remove that Set from: 
        • AI Chatbot Data Pool
        • “Unattended” Table View

      To mark an Answer as Irrelevant, select “Mark as Irrelevant” at the bottom of the Q&A Set’s drawer.

    What's Next