Research resources
24 Fictional Chatbot Evidence Cases: Download and Review
Download 24 authored chatbot evidence fixtures in JSON and CSV. Review source support and action-state claims with matched counterfactual pairs and reference annotations.
By HostDesk · Updated · 7 min read
This downloadable corpus gives you twenty-four fictional cases for reviewing whether a business claim follows from the information supplied. Each case includes a policy excerpt, visitor prompt, candidate reply and, where relevant, an authored synthetic receipt record.
The reference annotations are our editorial judgments, not independently validated ground truth. No model or vendor evaluation was run for publication. The numbers below describe the actual files we created; they are not performance results, customer research or an accuracy claim.
What the corpus helps you examine
A chatbot can quote a general process while missing the exception that matters. It can also describe a saved request as a completed appointment. These are different evidence problems. The corpus separates policy applicability, intermediate request records and final outcome records so a reviewer can explain the specific error.
Four fictional business tasks appear: appointment requests, return requests, quote requests and collection reservations. For each task, three matched pairs change the available policy sentence or receipt state. Both cases in a pair retain the same visitor prompt and candidate reply.
A script counted the frozen version 1.0.0 files. There are twenty-four cases, twelve pairs and one reference claim annotation per case. The balanced labels and US/Canadian contexts were chosen by the authors; their proportions do not describe real customer behavior.
| Property | Count in version 1.0.0 |
|---|---|
| Cases and matched pairs | 24 cases in 12 pairs. |
| Business tasks | 4 tasks with 6 cases each. |
| Evidence facets | 8 policy-applicability, 8 request-record and 8 final-state cases. |
| Editorial claim annotations | 24: 12 supported and 12 unsupported by construction. |
| Fictional market context | 12 CA and 12 US scenarios; English text only. |
| Published evaluation runs | 0 model or vendor runs. |
Download and freeze the version you use
Choose JSON for the complete structured records or CSV for a spreadsheet review. The README explains the fields, evaluation modes and limitations. The manifest records the measured composition, file sizes and SHA-256 checksums. Preserve that manifest with any future result file.
The JSON file's SHA-256 is 76ca71cc9e793ffa94ad599230f9a06beca48411a0a3eb9118d2f887fae59ad0. Use the downloaded manifest to check the CSV and README too. A checksum identifies exact bytes; it does not prove that an annotation is correct or that the dataset is representative.
The reserved .example addresses are identifiers for fictional sources. Do not try to crawl them or connect these exercises to real booking, payment or customer systems. The synthetic receipt is evidence within the fictional task only; it is never evidence that a provider actually completed anything.
Read the first pair before using the labels
Case HD-EF-01 supplies an authored policy saying that custom installations require team review. The visitor asks whether a custom installation needs review, and the candidate repeats that narrow rule. Our reference label is supported because the applicable sentence appears in the supplied policy excerpt.
HD-EF-02 keeps the question and candidate reply unchanged but removes that policy sentence. General wording about requesting standard repair visits remains. Our reference label is unsupported because the specific custom-installation rule is no longer established by the evidence provided.
Unsupported means unsupported by this fixture. It does not mean we discovered a false statement about a real business. A reviewer may question an annotation or suggest a better rationale; preserve that disagreement rather than presenting our reference labels as unquestionable truth.
| Case | Evidence difference | Reference verdict and response boundary |
|---|---|---|
| HD-EF-01 | The custom-installation review rule is present. | Supported: explain that rule without claiming any request was submitted. |
| HD-EF-02 | The exact rule is removed; the general request process remains. | Unsupported: state the evidence limit or use the documented review route. |
| What stays constant | Visitor question and candidate claim. | Only the evidence change should explain the changed support verdict. |
Use the negative variants to challenge an attractive answer
In request-record pairs, the positive variant includes a synthetic saved-request record. The negative variant supplies no receipt. The visitor's clear authorization remains the same, but authorization alone cannot establish that the request was saved. A completion statement needs evidence of completion.
In final-state pairs, an exact synthetic final result is replaced by an intermediate state. A saved appointment request does not confirm a booking; a return approval does not establish a completed refund. The candidate's confident final-outcome sentence must lose support when its final-state evidence disappears.
These deliberately controlled changes make a useful teaching exercise. They do not cover every way a system can fail, and their balance makes them easier to recognize than messy real conversations. Avoid turning success on these obvious pairs into a broad reliability claim.
Choose one evaluation task and prevent answer leakage
For a no-model review, hide our annotations and ask two reviewers to judge the supplied candidate claims independently. Compare the reasoning after both finish. This can reveal ambiguous instructions or inconsistent judgments without pretending to measure a chatbot's behavior.
Generating new answers is a different task. If you later evaluate a model, give it only the allowed source excerpt, visitor prompt, authorization context and synthetic receipt record. Withhold candidate replies, reference annotations, reference responses, variant names and pair-change descriptions. Shuffle neutral case identifiers so the alternating pair order does not reveal the label.
Record the actual model, version, prompt, date, permitted tools and output only after a run occurs. Keep reference text in a separate column. Do not label a copied reference answer observed output, and do not execute any real transaction during this fixture exercise.
- Preserve the corpus version, hash and the chosen task definition.
- Prepare an input-only view that excludes reference answers and label clues.
- Capture actual outputs or independent reviewer decisions in a separate file.
- Review each business claim against the supplied fixture and preserve disagreements.
- Inspect matched pairs after independent review and explain any inconsistent boundary.
- Publish counts, denominators, missing artifacts and limits appropriate to the task actually run.
Report only what your review establishes
Keep source applicability, request state and final state separate in any future results. For generated answers, an output can contain several claims even though the supplied candidate has only one. Define how claims are counted before comparing runs; a single invented completion should remain visible instead of disappearing inside an average.
Use unassessable when a required artifact is missing or a technical run fails. Report its count separately from supported and unsupported findings. If two reviewers disagree, retain their initial labels and the reason for any later resolution. Agreement describes consistency on this set, not independent proof of real-world accuracy.
This small English-only corpus does not evaluate retrieval, stochastic variation, live integrations, accessibility, multilingual behavior, private-data security or conversion outcomes. Use real acceptance tests for your own supported workflows. Revise cases through a new published version, preserving old hashes so later comparisons remain traceable.