Product evaluation framework for AI systems

Lesson 1.2 builds on the previous lesson's finding that AI systems produce different outputs on every run. That variance was quantified with pass@k and reliable@k. But knowing the system is unreliable does not reveal WHERE it is unreliable. Is the SQL wrong? Is the retrieval pulling bad context? Is the narrative fabricating numbers? Problems that cannot be located cannot be fixed. This lesson provides the map. The goal is to build an evaluation surface map: a complete inventory of every way an AI feature can fail, every way an attacker could exploit it, and every blind spot in current testing. By the end, the resulting artifact answers two questions a PM will inevitably ask: "What are all the ways this can fail?" and "What haven't we tested yet?"