A Chatbot Conversation Review Rubric You Can Use Weekly

Write the review question before selecting conversations

Choose a question that can lead to an operational decision. For example: are customers receiving the correct cancellation exception, or are requests reaching a usable next step? A general request to judge whether the bot is good produces inconsistent notes and makes the next review hard to compare.

Record the dates, assistant version, source version and included workflows. A conversation about a retired policy should be judged against the approved policy available at that time, while also checking whether the old policy should have been withdrawn. Preserve that distinction in your finding.

NIST's AI Risk Management Framework supports evaluation in context and continued monitoring during operation. The rubric below is our practical editorial method, not a NIST certification, prescribed scoring system or evidence that any particular assistant meets a standard.

• NIST AI RMF: measurement in context and during operation

Keep routine sampling separate from incident review

Create a list of eligible conversations before reading their contents. Exclude staff tests and obvious duplicates using written rules. Select a random sample from that list, or review the whole set when volume is manageable. Record how you selected it so a colleague can reproduce the selection.

Review complaints, unusual requests and suspected harmful answers as a separate incident set. Those cases deserve attention even when they are rare. Combining them with a random sample and publishing one failure percentage would overstate their frequency in the ordinary conversation population.

If you deliberately sample extra conversations from a language or workflow, report that stratum separately. A ten-conversation review is useful for finding defects; it does not support a precise population estimate. Keep unreviewed and unavailable records visible rather than silently treating them as passes.

Code the answer on separate dimensions

Read the customer's goal and the complete relevant exchange before judging a sentence. Then examine each business claim against the approved source. A correct general policy may be irrelevant to the customer's exception; a polite answer may avoid the actual question.

Use the labels pass, fail and unassessable for each dimension. Add not applicable only when the dimension truly does not arise, such as an action receipt in a purely informational exchange. Do not average a serious unsupported promise into a reassuring overall score.

A safe boundary can pass accuracy while failing usefulness if it offers no workable next step. Conversely, an effective contact link cannot rescue an invented refund promise. Keeping these dimensions separate makes the fix specific enough for the responsible owner.

Worked example: a correct fact with a missing exception

This example is fictional. A service policy at cedar-workshop.example/policies, version 3, says standard visits can be cancelled until the previous business day; custom jobs require team review. The visitor asks whether a custom job can be cancelled tomorrow. The assistant replies, 'Visits can be cancelled until the previous business day,' then offers the standard cancellation link.

The sentence quotes a real rule in the fictional fixture, but it omits the condition that matters. Mark factual support and goal addressed as fail: the applicable custom-job exception changes the answer. Mark outcome wording not applicable because the assistant did not claim a cancellation occurred.

The repair is to state that this custom job needs team review and offer the correct request route. Add the exception to the regression set. Do not record a saved customer, completed cancellation or lost sale: none of those outcomes is present in this example.

Use disagreements to improve the rubric

Have two reviewers independently code an initial subset before discussing it. Keep both original verdicts. Ask whether a disagreement came from different evidence, an ambiguous rule or a genuine judgment call. Revising a definition is more useful than pressuring the second reviewer to match the first.

For a simple agreement check, divide matching applicable labels by the labels both reviewers could assess. Report the numerator, denominator and rubric version. Agreement is a consistency diagnostic, not proof that both reviewers are correct. A policy owner may still need to resolve the underlying rule.

Restrict transcript access and redact details unnecessary for the review. Use references rather than pasting personal conversations into a broadly shared spreadsheet. If an excerpt cannot be safely shared, preserve a restricted evidence pointer and explain the review limitation.

Publish findings that lead to a recheck

Report counts by dimension with their own assessed denominators. Three unsupported replies among twenty assessable sampled conversations is different from three among twenty selected conversations when half lacked source evidence. List the unassessable cases and what would make them reviewable.

Give each material finding an owner, an approved change and a repeatable recheck. A content defect belongs with the source owner; a broken destination belongs with the site operator. After a fix, repeat the affected case and inspect new routine samples without rewriting the old verdicts.

For US and Canadian audiences, review the policies, currencies, service regions and languages actually represented in the sample. Do not infer a Canadian failure rate from US conversations, or vice versa. If a segment has no observations, report no observations.

• Build prelaunch acceptance cases
• Define a measurable pilot