How much evidence is enough to decide

You have 50 test queries with correct answers written ahead of time, and 42 of them come back right. That’s 84 percent. Your PM wants to roll out to everyone and asks whether 50 is enough. This lesson is how you answer that with a number instead of a guess.

Three things decide the sample size: the decision you’re making, how confident you need to be, and how noisy the measurement is. A full rollout needs a tight estimate, a limited ramp can live with a wider one, and comparing two versions needs a power analysis instead. Noise comes from the system and from the evaluator grading it.

The formula is n = z² × p(1 − p) / ε². You work through it on the 42-of-50 example against a rollout rule of plus or minus 5 points at 95 percent confidence, and see how far 50 samples falls short. Then you pull the two levers you control. Lowering the confidence level cuts the count. Tightening the margin raises it fast, because halving the margin roughly quadruples the samples.

An LLM evaluator isn’t perfectly consistent, and its kappa score tells you how much. The course uses a rough rule: divide the required sample size by kappa, so a noisy judge needs more samples to give you the same evidence.

The practice is a decision matrix: full rollout, hold and experiment, each at 90 and 95 percent confidence, with a sample size you work out from the formula and a two-sentence evidence argument in every cell. The extended version adjusts every cell for three evaluator kappa values.