Grounding evaluation in user value

Your dashboard says 92% SQL correctness, 87% retrieval accuracy and a 4.2% hallucination rate. The PM asks whether users find the AI Data Analyst useful, and you can’t answer, because none of those numbers has been checked against anything a user did.

This lesson splits metrics into three layers. Leading metrics like SQL correctness are fast and computed offline on a test set. Lagging metrics like edit rate, abandon rate and satisfaction come from real users in production. Business metrics like analyst time saved move slowest of all. A leading metric is a proxy, and it only earns a say in decisions once you’ve shown it predicts a lagging one.

You check that link with a correlation study on production traces, and the strength of the correlation decides the metric’s role. Above 0.7 it can block a release. Between 0.5 and 0.7 you track it and try to improve it. Below 0.5 you use it for debugging only. You predict which of three candidate metrics tracks satisfaction most closely, see an example result, and walk through one trace where the query ran, rows were missing, and the user was still satisfied. You also see metrics that are easy to log and predict nothing, and what happens when a team optimizes one of them.

The practice is a correlation table and a Leading-to-Lagging Metric Map. You put the six example metrics in one table with their correlations, intervals and tiers, decide which one could block a release, and fill in the map for one metric from your Lesson 1.3 failure taxonomy.