Test set strategy and dataset lifecycle

This is Lesson 5.2, Dataset Lifecycle. In the previous lesson, the evaluation pipeline was built with six stages and a warehouse that stores every run with full metadata. That is a real achievement. But here is the question this lesson asks: what happens when the data flowing through that pipeline stops representing reality? All of that infrastructure assumes the evaluation dataset is valid, meaning it reflects the kinds of queries users actually ask. If the dataset drifts away from production, every result in the warehouse is measured against the wrong baseline. The metrics look great. The users have a different experience. This lesson covers the four-stage lifecycle framework that keeps evaluation datasets valid as the product and users evolve. Creation, development, test set management, and retirement: each stage has explicit transition gates. The lesson explains why splitting data before any metric work is non-negotiable, why versioning matters as much for datasets as it does for code, and how to detect when a test set has drifted too far from production. The five governing rules exist to prevent two failure modes: teaching to the test, where optimization of the judge for specific examples causes a loss of generalizability; and staleness, where the test set no longer matches what users actually do.