Evaluation pipeline architecture and environments

An evaluation that someone runs once and pastes into Slack can’t be compared with anything. If SQL correctness reads 87 percent on Monday and 83 percent on Wednesday, you can’t tell whether the system got worse or the sample, the judge or the dataset changed, because none of that was saved.

This lesson sets out the structure that fixes that. Every run goes through six stages: sampling, judging, aggregation, storage, reporting and alerts. Every run also writes a record of how it was done: the dataset version, model version, judge version and configuration, code commit, sample size and sampling strategy. With that record you can look at two runs and say what differs between them. A worked example shows why: two runs on the same traces that differ only in the judge version can’t tell you whether the system improved.

You also see where the v1 AI Data Analyst actually breaks, using the first failure step each trace records, and how much evaluation to run at each stage of a rollout, from the full suite offline to sampled blocking metrics with alerts in production.

The practice is on paper. You write the run record for a baseline run, use the failure counts from the lesson to explain the top two failing steps, explain the gap between the two runs in the worked example, then design a third run that changes one thing on purpose and predict what it will show. In Lesson 5.7 you run the whole pipeline by hand.