Driver analysis: explaining variance and choosing what to change first

The AI Data Analyst scores 74 percent SQL correctness on its test set, just over a 70 percent blocking threshold. Your PM wants to know whether to release now or spend two more weeks on it. The overall number can’t answer that, because 74 percent looks the same whether the failures are spread evenly or packed into one kind of query.

Driver analysis splits the overall number into what each segment contributes. The impact score is the segment’s gap from the overall rate times its share of queries, so a segment that looks terrible but is 2 percent of traffic can rank below one that looks mild but is 30 percent. You sort by that score, and the top of the list is where engineering time does the most.

The findings are associations. A group of users can look worse only because they ask harder questions. So you split by two dimensions at once before blaming one factor, and you treat every finding as a hypothesis for the Week 5 experiments.

In the worked example you predict how simple and complex queries compare, then split the same test set by domain and by domain and complexity together. That second split shows an interaction effect: complexity hurts everywhere, and much more in one domain.

The practice works from the lesson’s own tables. You score the domain and each domain and complexity cell by impact, rank them, and fill an intervention hypothesis for the worst cell: the likely cause, the fix, the expected gain, and how you’ll check it worked. On the extended track you work out what the overall rate becomes if that cell gets fixed, and plan how you’d check the drivers month to month.