Cost-aware evaluation on a fixed budget

Once some of your metrics are LLM judges, running every metric on every query gets expensive fast. In the lesson’s scenario, the AI Data Analyst handles 10,000 queries a day and costs about $3,000 a month to run, while evaluating everything costs about $33,300 a month. Two judge metrics make up about 90 percent of that. Leadership asks for a 90 percent cut without losing sight of quality.

The obvious answer is to sample 10 percent of everything. That treats every query as equally important, so rare and costly failures like PII leaks are mostly missed while routine traffic gets the same coverage as the dangerous queries.

The lesson uses three strategies instead. Stratified sampling sets coverage per traffic segment based on what a missed failure would cost, with full coverage on safety-critical traffic. Blocking metrics, the ones that gate the release, get the budget first. And a judge cascade runs free rule checks and a small, cheap judge before sending only the unclear cases to the expensive frontier judge.

You cost out each step with a simple formula, working backward from the budget to the coverage you can afford, and you predict whether stratified sampling alone gets under budget before you see the answer.

The demo’s cascade still sends a quarter of narratives to the expensive judge. In the practice you pick a split that sends fewer than a fifth, work out the new monthly bill with the same formula, say what you’d check before trusting it, and write a Cost Allocation Plan with the coverage matrix, total cost, monitoring and what would make you expand coverage.