Scaling semantic evaluation with statistical confidence

In L3.5, an LLM judge was built and its agreement with human labels was validated using Cohen's Kappa. That was the first question: does the judge agree with humans? Now comes a harder question. The one that really matters. Given that the judge is imperfect (and the Kappa score confirms it's not perfect) what is the true pass rate of the system? And beyond raw accuracy, how can systematic biases that corrupt judge scores be detected even when overall accuracy looks good? Position bias and verbosity bias. Two silent killers. By the end of this lesson, a Judge Report Card will characterize the measurement instrument completely. A feature wouldn't ship based on a broken analytics dashboard. Don't ship based on an uncalibrated judge. Here's what this lesson builds toward. Three things are needed. One: the corrected pass rate that accounts for judge error. Two: bias detection results that indicate whether the judge is systematically wrong in specific scenarios. Three: confidence intervals that indicate how much to trust the corrected number. All three go into the Judge Report Card, the calibration certificate for the measurement instrument. That's the artifact built today.