Monitoring for drift and regressions

Say v2.1, the fix from 5.4, clears its rollout and is live. The evidence behind that decision starts to age. Users ask different questions, someone renames a column in the warehouse, the model behind your judge gets updated. Say the system handles 50,000 questions a day and you can afford to evaluate about 100, so the lesson is about which 100 to look at and what else to watch.

You weight the daily sample toward traces with warning signs: negative feedback, low confidence, question types that have been hard before. The weighted sample passes at a lower rate than a random one, which is what you want, and you report both numbers with what each one answers.

Then four checks, one per kind of drift. A stability index on the mix of question types for input drift. A distribution test on features of the generated SQL for output drift. A sentinel set of questions with known answers, run every week, for concept drift. And agreement between the judge and human labels on a fixed set, for judge drift. The sentinel example shows why a slow decline needs a check against the baseline: a week-over-week alert misses it because each step is small.

The practice is a monitoring plan for v2.1: sampling budget and weights, a detection method for each drift type, tiered thresholds, a table of who acts on which alert and how fast, and the response workflow.