The Chat Quality tab automatically scores the replies Mira writes to guests in her own words and lists the problem ones for you. It exists to answer a single question: "How reliable are Mira's replies, and where does she stumble?" This page explains what every number and label on the screen means and how to fix a problem.
What is on the screen#
In the left menu, under the Analytics heading, click Chat Quality (the icon is a shield with a check mark). From top to bottom the page is laid out like this:
- The Chat Quality heading and, below it, the description "The self-evaluated quality score of the AI's free-text replies."
- Four summary cards (side by side on a wide screen, stacked on a phone).
- A short info box titled What are the evaluation criteria?
- The Flagged Replies list: "The most recent replies that triggered at least one quality criterion."
The page is entirely read-only: you do not save, delete or mark anything here. You make the fix on the pages the reply came from (explained below).
Summary cards#
| Card | What it shows | Note |
|---|---|---|
| Evaluated Replies | The total number of Mira replies scored so far | Only replies that are evaluated are counted (see "Which replies are evaluated" below) |
| Average Confidence | The average confidence score of all evaluated replies, as a percentage | Shows 0% if there are no replies yet |
| Flagged Replies | The number of replies that triggered at least one quality criterion | Red card |
| Clean Rate | The share of evaluated replies that triggered no criterion at all | (Evaluated − Flagged) ÷ Evaluated |
The evaluation criteria box#
The box under the cards sums up the criteria in one sentence: each reply is evaluated from its own text, with no extra AI call. So this feature adds no cost and does not make the guest wait longer for a reply. The criteria are explained in detail below.
The Flagged Replies list#
The list shows the 100 most recent replies that carry at least one flag, newest first. Each row shows:
- the guest's name and, if available, their phone (Unnamed guest if there is no name),
- the date and time of the reply,
- on the right, an amber Confidence: badge (the percentage score of that reply),
- on the right, a View Conversation button that opens the detail page of that conversation,
- small red flag badges (one for every criterion the reply triggered),
- underneath, the first 220 characters of the reply (with "…" at the end if it is longer).
The invisible marker Mira adds to a reply to call up the contact card is not shown in this preview; you only see the text the guest read.
Empty states#
- If no reply has been evaluated yet, you see No evaluated replies yet. Old replies written before this feature was added are not scored retroactively; the list fills up as guests chat.
- If replies have been evaluated but none carries a flag, you see No flagged replies: "None of the evaluated replies currently trigger any quality flag."
Which replies are evaluated#
Some of Mira's replies are produced directly from the records in your panel with a ready-made template (prices, availability cards, Facilities, Extra Services, Business Info fields, the reservation flow, a Knowledge Base answer that matches exactly one entry, greetings and so on). These replies come from data you wrote, so they carry no risk of invention and are not scored.
Two kinds of replies are scored:
| Reply type | When it happens | Note |
|---|---|---|
| Free-text reply | Mira writes the answer in her own words, using your Knowledge Base and business information | All six criteria are checked |
| "No verified information" reply | The guest asked something about your business, but nothing in your panel matches it; Mira showed the contact card | Always gets the Missing info flag. It is included on purpose, because for you the real question is "did the guest get a real answer?" |
The scope reply to off-topic questions ("What is the capital of Turkey?") is not scored; that is not a failure but a deliberate limit.
The criteria one by one#
Every reply starts at 100% confidence. For every criterion it triggers, the points below are deducted. A reply can trigger several criteria at once; the score never drops below 5%.
| Flag (in the panel) | When it is set | Deduction |
|---|---|---|
| Missing info | Mira said "I do not currently have verified information about this" or showed the contact card (Call Us / WhatsApp / Send an Email) | −30 |
| Translation issue | Mira wrote the reply in a language other than the guest's (e.g. Turkish to an English-speaking guest); the system tried to translate the reply but failed, and the guest got the contact card instead of a translated reply | −40 |
| Too short | The reply is shorter than 12 characters after leading and trailing spaces are removed | −30 |
| Too long | The reply is longer than 1600 characters | −15 |
| Hedging language | At least two different hesitation phrases appear in the reply. Turkish: "sanırım", "galiba", "emin değilim", "olabilir", "tahmin ediyorum"; English: "i think", "i'm not sure", "might be", "probably", "i guess"; German and Russian equivalents are checked too | −20 |
| Repetitive text | The same sentence of at least 20 characters is repeated word for word three or more times in the reply | −30 |
How to read the confidence score#
| Score | Meaning |
|---|---|
| 100% | The reply triggered no criterion. It does not appear in the list. |
| 85% | Only a long reply. Usually not a problem. |
| 80% | The reply contains hesitation phrases. This can mean the information is unclear. |
| 70% | One serious problem: most often Missing info. |
| 30% and below | Several problems together. Look at these first. |
This is an approximate measure: the rules look at the text and cannot tell whether the content of the reply is correct. What it does catch reliably: questions where the guest was left without an answer, replies sent in the wrong language, and broken, self-repeating text.
Quality information in the conversation detail#
The same evaluation also appears on the Conversations page. When you open a conversation, under every scored Mira reply, on the right, you see:
- an AI • Confidence: line (that reply's score),
- if there are flags, small amber flag badges (Missing info, Hedging language and so on).
These lines do not appear on template replies or on messages written before this feature. For details, see the Conversations page.
Flag labels are shown in the panel language (Turkish or English). The language the guest speaks (TR/EN/DE/RU) is independent of this; the evaluation works in all four languages.
Step by step: Fixing a flagged reply#
- Open the Chat Quality tab and scroll down to the Flagged Replies list.
- Start with the rows that have the lowest Confidence: score, or where the same flag shows up again and again.
- Press the View Conversation button on the row. Read what the guest asked together with Mira's reply; the list only shows the reply, you see the question in the conversation.
- Use the table below to find the right page for the flag, and add or correct the information.
- Test it yourself: open Widget → Guest Link and ask the same question (in the same language if needed). Check in the Chat Quality list or in the conversation detail that the new reply has no flag.
What to do for each flag#
| Flag | Likely cause | Where to fix it |
|---|---|---|
| Missing info | The topic is not in your panel at all, or the requested detail (time, price, distance) is not written in the entry | For fixed topics (check-in/check-out, pets, cancellation, Wi-Fi, contact) use Business Info; for facilities such as a pool or breakfast, Facilities; for paid services, Extra Services; for everything else, the Knowledge Base. The easiest route: Knowledge Base → Missing Topics |
| Translation issue | Mira tried to pass on an entry written in your content language without managing to translate it into the guest's language | Rewrite the related Knowledge Base entry in short, simple sentences; check that Business Info → Regional Settings → Content language is set correctly. If it was a one-off outage, asking the same question again fixes it |
| Too short | The entry the reply relies on is very short or a single word | Write the entry as a full sentence: instead of "Yes", "Yes, our pool is open from 08:00 to 20:00." |
| Too long | A Knowledge Base entry covers several topics, or many entries match the same question | Split long entries into short, single-topic entries |
| Hedging language | The information in the entry is vague ("usually", "depending on the situation"), or two entries contradict each other | Write clear information; switch one of the contradicting entries off with Turn Off AI Usage or delete it |
| Repetitive text | A rare glitch of the model | A one-off is not important. If it keeps happening for the same question, simplify the related entry and let us know at support@airoomier.com |
How it relates to Missing Topics#
Chat Quality and Knowledge Base → Missing Topics show the same problem from two different angles:
| Missing Topics | Chat Quality | |
|---|---|---|
| What it lists | The questions Mira could not answer | Mira's problem replies |
| Grouping | The same question on one row, with a N time(s) asked counter | Every reply on its own row |
| Scope | Missing information only | Six criteria (missing info, translation, length, hedging, repetition) |
| Actions | Add to Knowledge Base, Mark Resolved, suggestions prepared overnight | Read-only; View Conversation |
In practice, most replies flagged Missing info have a matching row in Missing Topics. So the recommended way of working is:
- Close information gaps from Missing Topics: there the questions are grouped, the most frequently asked is at the top, and they come with a ready draft title.
- Use Chat Quality to find the problems Missing Topics does not show: Translation issue, Hedging language, Too long, Repetitive text. These are cases where Mira did give an answer, but the answer was not good.
Differences between the two lists:
- Replies that showed the contact card because of a Translation issue also get the Missing info flag, but they are not written to Missing Topics.
- If a question where Mira said "I have no information" in a free-text reply is classified as off-topic, it is not written to Missing Topics, but the reply is still listed with the Missing info flag.
- Closing a topic with Mark Resolved in Missing Topics does not remove old flagged replies from Chat Quality. The list shows history; you see the effect of your fix in new replies.
Who can see it, plan restrictions#
- The Chat Quality tab is the same on the Basic and Pro plans; there is no separate limit and no Pro-only section.
- Every user except the Staff role sees this page and can open it; Staff users only see the Guest Requests board (Team and Roles).
- If you have not finished the setup wizard, this tab also sends you to the wizard first (Getting Started).
- Because the evaluation makes no extra AI call, it does not use up your monthly AI quota either.
Example scenario: a 30-room hotel#
You run a 30-room hotel in Antalya. At the end of a week, the Chat Quality tab shows: Evaluated Replies 120, Average Confidence 86%, Flagged Replies 24, Clean Rate 80%.
Scrolling down to the list:
- 14 rows have only Missing info, all with Confidence: 70%. With View Conversation you see that guests asked about "spa", "massage" and "parking fee". In Knowledge Base → Missing Topics the same topics are already there with 5 time(s) asked and 4 time(s) asked counters. You add a Facility for the spa and massage and a Knowledge Base entry for the parking fee, then close the topics with Mark Resolved.
- 4 rows have Translation issue and Missing info together (Confidence: 30%). They are all German-speaking guests asking about "Frühstück". You notice that your breakfast information sits in a long, single-paragraph Knowledge Base entry. You move it into a Facility called "Breakfast" in short sentences.
- 5 rows have Hedging language: Mira answered late check-out questions with "I think it might be possible". You make the Knowledge Base entry "Late check-out can be arranged depending on the situation" clear: "Late check-out is free until 14:00; for later times, please ask reception."
- 1 row has Too long; it is a one-off, so you ignore it.
The following week, as new replies come in, Average Confidence and Clean Rate go up; the old 24 flagged replies remain in the list.
Tips and common mistakes#
- "I fixed it, but the flagged reply is still in the list": This is normal. The list shows past replies, and no reply is re-scored later. Ask the same question again from the Guest Link and look at the new reply.
- "Evaluated Replies is much lower than the number of messages in Conversations": This is normal. Price, availability, facility, service and reservation replies are produced from templates and are not scored. The better your panel is filled in, the more template replies Mira gives.
- A high Average Confidence does not mean everything is fine. The rules do not check whether the content of a reply is correct. Keep reading conversations from time to time.
- Fix the repeating ones first. If the same topic appears on several rows, adding a single entry fixes dozens of future replies.
- Close missing information from Missing Topics. There the questions are grouped and come with suggestion drafts; it is faster for this job (Knowledge Base).
- For translation issues, simplify your entries. Short, clear sentences translate more reliably. For details on working in several languages, see How Mira Works.
- Look once a week. A weekly look at this tab together with Missing Topics keeps Mira's reply quality consistently high.