Deriving evaluation signals from available ground truth

By Week 3 the AI Data Analyst logs something at every stage: retrieval scores, the SQL it wrote, execution results, the summary text, latency and token counts. The PM asks whether v1 is ready to ship, and a dashboard full of numbers still doesn’t answer that. What’s missing is a decision, made ahead of time, about which numbers you measure and what each one is for.

An evaluation signal is the number you get when you compare the system’s output to ground truth, the known right answer. Every signal has a type and a role. The type is how you compute it. Execution-based oracles run the output, for example running the generated SQL and a reference query and comparing the result sets. Structural signals check format, like whether the JSON parses or the required fields are there. Semantic signals need a judge and a rubric, and the judge brings its own variance on top of the system’s. The role is the decision the signal feeds. A gate blocks a release, a diagnostic helps you find where a failure came from, and a driver explains where quality varies. If you can’t name the decision a signal feeds, you drop it.

You take an example hit rate on 300 test queries, work out how many queries reach the SQL step with nothing relevant, and look at what a retrieval miss does to the SQL and the summary that come after it.

The practice is a signal catalog. You work out what Precision@5 and MRR would add to the hit rate, describe what over-retrieving and under-retrieving look like, and write down at least five signals with their type, what they’re checked against, their role, their cost and how they’re computed. The extended version adds a judge-based signal with its variance written down and grows the catalog to ten signals.