Failure surfaces and annotation-based analysis

Say fifty complaints arrive in seven days and no two are described the same way. One user got last quarter’s data when they asked for this quarter. One had SQL fail with no error message. One read “slight decline” in a summary when revenue dropped 40%. Your instinct is to reach for a checklist, and a checklist only catches the failures somebody already imagined.

So you go the other way. Read the traces with no categories. Write one note per trace in your own words. Cluster the notes and let the categories come from the data. Keep reading until new traces stop producing new categories. Then triage every category with one of three labels: prompt-fix, evaluator-needed, or system-fix. The label tells you where the work goes.

The step people get wrong is saturation. Ten traces is not enough. Twenty is the minimum and thirty is better. The check is simple: read ten more, and if a new category appears you are not done.

You read the first five traces in the v0 file yourself. Answers about a different metric than the one asked. A summary that calls a rise from 31,981 to 48,040 a -4.5% change. A retention rate of 101671%. Most of them don’t fit “hallucination” or “SQL error” cleanly.

The practice is a failure taxonomy built from your notes on those five traces: named categories, each with a severity and a triage label, and the saturation rule you would use on the full file. Write your own categories before you look back at the worked answer.