Metric design patterns for AI features

The AI Data Analyst has a score for retrieval, one for SQL and one for the narrative, and you still can’t tell your PM whether it’s ready. Each metric was built by whoever owned that component, and no one decided which one can hold a release or what level to measure at.

This lesson gives you a starting point for each metric instead of a blank page. A measurement archetype is a reusable plan for one type of AI feature. It sets the unit of measurement, the kinds of quality that matter, the blocking and optimization metrics, and how scores roll up. Six archetypes cover most features: drafting, summarization, extraction, RAG, agents and decision support.

The AI Data Analyst is a hybrid, so each component gets its own archetype. Retrieval follows RAG and is measured on its own. SQL generation is checked by running the query and comparing the result with a known answer. The narrative follows summarization. You predict which unit of measurement fits the narrative, see why one end-to-end score can’t tell you which component to fix, and see why SQL correctness blocks the release while narrative conciseness gets tracked.

The practice fills in the archetype template for the AI Data Analyst. You pick the part of the question whose extraction errors would hurt most, justify a precision bar for refusal detection, and work out a ten-question session’s pass rate two ways, worst case and average, from the lesson’s 89 percent SQL correctness. The two land far apart.