Measure a Website Chatbot Pilot Without Inventing ROI

Write one decision the pilot should inform

A useful question is whether offering an assistant on a defined service page increases qualified enquiries without increasing unsupported promises or staff cleanup. Write the business definition of a qualified enquiry before looking at the pilot results. An enquiry might require a supported service, a usable contact method and enough visitor-supplied detail for the team to respond.

Choose one primary outcome, a fixed observation window and a decision date. Keep purchases, enquiries and booked appointments separate unless your business has a defensible common outcome definition. A submitted request is not a booked job, and a booked job is not recognized revenue.

Microsoft Research's experimentation guidance emphasizes a testable hypothesis, appropriate randomization and several types of metrics, including guardrails and data quality. The ledger below applies those principles without promising that a small business pilot will have enough traffic to detect a commercial effect.

• Microsoft Research: trustworthy pre-experiment design

Define the eligible population and counting unit

Count everyone assigned to the pilot experience who meets the prewritten eligibility rules, including people who never open chat. Comparing chat users with nonusers answers a different question because people choose whether to start a conversation. Assignment must happen before that choice if you want to estimate the effect of offering chat.

Use a stable counting unit supported by your measurement design. A browser identifier is not a guaranteed unique person: devices, deleted storage and restricted tracking can change it. If you can only measure sessions, call them sessions and acknowledge repeat visits. Do not label that denominator people.

For US and Canadian traffic, record the country signal's source and unknown category. Preserve USD and CAD separately in any money fields. Choose the reporting timezone and outcome window explicitly so a late evening enquiry or delayed team qualification is not assigned inconsistently.

Choose a comparison you can defend

Where feasible, randomly assign eligible units to the usual site or the site with the assistant, and keep the assignment stable during the observation period. Preserve ordinary contact paths in both groups. Check that assignment and outcome records join correctly before interpreting any difference.

A simple before-and-after comparison can be useful operationally, but seasonality, advertising, price changes and customer mix can explain the movement. Record those changes. Do not call the difference incremental chatbot revenue when the design cannot isolate the assistant's contribution.

Choose duration and sample needs before launch with appropriate statistical help when making a causal claim. A small result with a wide range of plausible effects may justify more learning rather than a verdict. Repeatedly checking results and stopping the moment they look favorable undermines a conventional fixed-horizon test.

Keep engagement, outcome and guardrail metrics separate

Record the funnel so you can diagnose where a change occurred. Assignment, assistant availability, chat opening, a supported answer, an enquiry and a qualified outcome are different events. A click toward an external booking system should remain a click unless an authorized record confirms the booking.

Track quality and operational cost alongside the primary outcome. Include unsupported commercial claims, failed handoffs, complaints, staff review time and service interruptions. These need clear counting rules too. An improvement in enquiries may be unacceptable if it depends on inaccurate promises or overwhelms the receiving team.

Do not add metrics with different units into one conversion total. Report outcomes as counts and rates, staff time as minutes or hours, and money with its currency and cost definition. Revenue, contribution margin and avoided labor are different quantities.

Worked example: why the denominator changes the story

These are invented figures for calculation only. Suppose 1,000 eligible browser identifiers are assigned to each group. The usual site produces 20 identifiers with qualified enquiries; the assistant group produces 24. Of the assistant group, 100 identifiers open chat, and 8 of those have a qualified enquiry.

The primary observed rates are 20/1,000 = 2.0% and 24/1,000 = 2.4%. Their difference is 0.4 percentage points; the relative difference is 20%. The 8/100 = 8% rate among chat openers describes a self-selected subset and cannot be substituted for the assigned-group comparison.

These counts alone do not establish a statistically reliable improvement, revenue or profit. An analyst must assess the design, uncertainty and complete follow-up window. The eight chat-associated enquiries also belong inside the twenty-four total; adding them again would double count outcomes.

Make the decision with limitations attached

Before reading the result, write the conditions that would stop the pilot immediately, such as a material false promise or exposed private information. Write separate conditions for pausing analysis, such as missing outcome records or broken assignment. A data problem is not a poor conversion result; it is a reason to repair measurement.

At the planned review, publish the population, counts, rates, uncertainty, guardrails and known changes. If the evidence is inconclusive, say so. Qualitative conversation review and observed customer tasks can still identify useful repairs while the business accumulates enough outcome data.

Treat source preparation, analytics configuration and any actual integration as separate work to verify. A public preview can help inspect published answers; it cannot establish incremental sales or replace outcome records from your business. Keep the plan so later pilots use comparable definitions.

• Review conversation quality separately
• Prepare a controlled launch