Metric strategy: blocking metrics vs optimization metrics

By Week 4 the AI Data Analyst has twelve evaluation metrics, and your PM still can’t get an answer to whether v2 should go out. Twelve good numbers don’t make a decision on their own, because the team hasn’t said which ones matter for a release.

This lesson gives every metric one of two jobs. A blocking metric has a threshold, and if it misses, the release waits. An optimization metric has a target, and missing it starts work without holding the release. To sort them you ask one question of each metric: if this one fails and everything else passes, do we release?

The metrics also get a shape. User trust sits at the top, tracked through a proxy. Four drivers sit under it: answer correctness, response latency, cost per query and multi-turn coherence. The metrics you compute from traces sit under those. When a driver drops, the tree tells you where to look first.

Every blocking threshold needs a written reason, meaning a sentence on what breaks below that number. The reasons usually come from one of four places: the output has to work at all, a cost or revenue limit, what users already have today, or the point where people give up. You predict how many of the twelve metrics should be blocking, then compare your number with a worked classification.

The practice is a metric system spec. You classify four metrics that are hard to call, write release criteria for a v2 retrieval change, and, on the extended track, put numbers on your threshold reasons and test one tradeoff: recall up 5 points, latency up 20 percent.