Evaluation cadence and governance

Last lesson put one owner on every evaluation activity. This lesson sets how often those owners look. The example is a SQL correctness judge on the AI Data Analyst that drifted for six months after launch. The team knew how to calibrate it. The monthly review that would have caught the drift got cancelled twice and never came back, and nobody had written down that the work was deferred.

The cadence has three tiers, because problems show up at different speeds. A weekly check of about 45 minutes catches acute problems like an error spike or a failed must-pass metric. A monthly review of 90 minutes catches slow degradation: trends over 30 days, a judge calibration check on 20 real queries, segments getting worse, and the evaluation debt. A quarterly audit of half a day refreshes the regression suite, recalibrates the judge on a larger held-out set, and asks whether the metrics still match what users need.

Each review gets one Accountable owner in a RACI matrix. Work that gets deferred goes in an evaluation debt register with a risk level and a paydown date, reviewed every month and capped at 10 open entries.

You replay the drift with the monthly review in place, work out whether a five-person team can afford the cadence, and see how the speed of detection changes how much uncertainty a ship decision can carry.

The practice is your own Evaluation Cadence Package: a cadence calendar of 11 activities across the three tiers, a RACI for six activities with one A per row, and three debt register entries with at least one High. The extended version adds a three-level escalation playbook and a simulation of skipping the monthly review for two months.