Test set strategy and dataset lifecycle

Say a test set passes 78 percent of 200 curated cases, and two weeks after release production quality is at 52 percent. The cases were six months old and mostly simple lookups, while users had moved on to multi-table joins and trend comparisons. Evaluation data goes bad in three ways: it goes stale, it gets contaminated when test examples leak into development, and it loses its version history so no one can reproduce an old result.

The lesson follows a dataset through four stages. At creation you split it into train, dev and holdout, 10, 45 and 45 percent, before any judge or metric work. In development you iterate on train and dev only, and every change, label corrections included, gets a version and a changelog entry. Once the judge settles, dev becomes a locked regression suite that you rotate on a schedule. When it no longer fits production, you retire it into an archive and keep it, because past release decisions cite it.

A large part of the lesson is keeping the holdout independent. Running it after every prompt revision, copying its examples into prompts, or tuning toward what you heard it contains all leak information into development. You predict what happens to a judge’s agreement score when it finally meets the holdout, and see why the dev-set number is too high.

The demo shows how to check whether a suite still matches production, with a Kolmogorov-Smirnov test on query complexity, user role and failure category, and how to decide whether to rotate.

The practice is a Dataset Management Spec for the AI Data Analyst’s regression suite, written on paper: you work out the split sizes, write the version history, set the drift check and what triggers a rotation, and write the holdout rules and retirement criteria.