Metric strategy — blocking metrics vs optimization metrics

This is Lesson 4.1: Metric Strategy. The past three weeks covered building individual evaluation metrics: retrieval recall, SQL correctness, semantic judges, bias detection, calibration pipelines. Each one measures something real about the system. But here is the problem: having twelve good metrics does not mean a team can make a ship decision. More numbers often lead to more arguments and more confusion about what actually matters. This lesson answers the question every PM eventually asks: which metrics matter for ship decisions, and which are just nice to track? By the end, a decision framework will turn a flat list of twelve metrics into a structured system. Some metrics gate releases; if they fail, the feature does not ship. Others track improvement but do not block deployment. That classification changes how thresholds are set, how monitoring works, and who owns each metric. The key concept: not all metrics have the same job. A metric that indicates the system cannot return results is fundamentally different from one that says the narrative could be more concise. Treating them the same is evaluation theater.