Experiment design for stochastic systems

The previous lessons covered building offline evaluation metrics and curating test datasets. Now the question is: can this change be shipped to real users? Here is the gap. Offline metrics describe what happened on a test set. They do not describe what happens when real users query the system in unpredictable ways. Real users ask follow-ups. They reformulate when confused. They abandon when trust breaks. They paste unexpected data into the input field. None of that shows up in offline metrics. Consider the implications: a beautiful evaluation pipeline was built and run, the numbers looked great, and none of it captures what actually happens in production. This lesson teaches how to design experiments for AI systems, meaning systems where running the same query five times produces five different results. That randomness is what "stochastic" means. The experiments must account for that variability, so that ship decisions are backed by evidence, not hope.