Experiment design for stochastic systems

A better offline score for v2 doesn’t tell you what happens when real users get it. They ask follow-ups, rephrase when they’re confused and give up when an answer looks wrong. To find out, you run an online experiment, and an AI system makes that harder than a normal A/B test. The same question can come back with a different answer on every run, so there’s more noise and you need more users to see the same effect.

The lesson walks through the design decisions you make before any data comes in: what you’re measuring, what gets randomized, the primary, secondary and guardrail metrics, the sample size from a power analysis, the stop conditions, and the rules for each outcome: ship, ramp, hold or roll back. Writing the rules first keeps you from bending them once you’ve seen a number you like.

Then you read the course’s own results. v2 raised SQL success by 2.7 points, a real gain, and it also raised latency 17.6 percent and cost 20 percent, past the 10 and 15 percent guardrails set before the test. By the rules, that’s roll back.

It also covers what happens when users share something. The AI Data Analyst has a shared cache, so control users can pick up some of v2’s better retrievals. A switchback, where everyone gets the same version in each six-hour block, is meant to remove that. On the course data it came back smaller and much noisier, with an interval that crosses zero and a carryover gap of its own, so it doesn’t change the call.

The practice is an Experiment Design One-Pager for the v1 to v2 change, built from the numbers in the lesson: check the sample size against the power analysis, match the effect and interval to a rule, work out both guardrail changes yourself, and write the decision. The extended version adds the switchback and carryover results and designs the rerun on the fix.