Ground truth sources, regression suites, and synthetic data

Your AI Data Analyst scores 90% on SQL correctness. The first question to ask is what that 90% was measured against. If the expected answers were copied from an older model’s output and nobody ran them against the database, the suite can pass while users see wrong numbers.

Ground truth is the set of answers you have checked and trust. It comes from different places for different tasks. For SQL you use an execution oracle: run the generated query and a verified expected query against the warehouse and compare the result sets, so two differently written queries that return the same rows both pass. For the written summary you need people reading it against a rubric, and before you scale that you check that two annotators agree, using Cohen’s kappa. User feedback is useful for watching production and too noisy to write test cases from.

Verified cases go into a regression suite, the set every release has to pass. You start with 10 to 20 high-impact cases from your Week 1 failure taxonomy and add more through promotion. A case gets in only if it is reproducible, covers a failure mode the suite doesn’t already cover, has verified ground truth, and is severe or frequent enough to be worth maintaining. For gaps production hasn’t shown you yet, you generate synthetic queries from a set of attributes and drop the ones no real user would ask.

The practice is a Regression Suite Promotion Plan: write the oracle’s explanation for the worked Q4 failure, run three failures from your taxonomy through the promotion criteria, and list three coverage gaps. The extended version writes synthetic queries for one gap and applies the realism check.