Semantic metrics with human and model judges

The SQL is right and the chart is right, and the summary still tells the PM that Q4 revenue grew 12% when the data says 8%. Retrieval metrics and SQL checks both pass, because the mistake only exists in the words. To catch it, something has to read the summary against the data.

At a thousand questions a week, that something is an LLM judge: a model that gets the question, the query results, the summary and a rubric, and returns Pass or Fail. You design one for narrative faithfulness. The rubric is binary, because a 1 to 5 score drifts between runs. It spells out a 5% materiality threshold and the edge cases. The judge writes its reasoning before the verdict, so you can see why it disagrees with a person.

Then you check it against human labels. You split about 100 labeled summaries into train, dev and test, and measure agreement with Cohen’s Kappa, which corrects for the agreement you’d get by chance. You read an example first run on 10 summaries where the misses all fall the same way, and use that pattern to decide what to change in the prompt. The course target is a Kappa of 0.7. Below 0.6, the judge’s numbers shouldn’t support a ship, ramp, hold or roll back call.

The practice is one round of refinement on paper: rewrite the Fail criteria to catch the misses, work out Kappa for a rerun grid, write the log entry, and set a stop rule. The result is a Semantic Rubric with Calibration Plan: the rubric, the few-shot examples, dev and test Kappa, and the cases where the judge still gets it wrong.