What to log and why it matters for AI evaluation

Week 2, Lesson 1 opens with a question every product team eventually faces: "What data do you need us to log?" Last week covered failure mode discovery in the AI Data Analyst, tracing through all the ways it could break. This week answers the instrumentation question: what needs to be logged to make evaluation possible? The stakes are high. This is the contract between product, evaluation, and engineering: the agreement about exactly which fields get logged. Get it right and measurement is unlocked. Get it wrong and Week 3's exercises will fail because the data doesn't exist. Engineering implements this in one sprint, but it's not reversible. The traces not logged today are gone forever. This is the lesson that separates "we should evaluate" from "we CAN evaluate." Evaluation, in this context, means measuring whether an AI feature is actually working, which is what this entire course builds toward.