Designing instrumentation that scales with the product

Lesson 2.3 built full instrumentation: spans, tool calls, intermediate outputs, confidence signals. Everything needed for debugging and evaluation. Full instrumentation. Zero compromises. And it works great in development. In staging. In a demo environment with 50 test queries. Now it's going to production scale. 100,000 queries per day. At that volume, logging everything on a managed observability platform becomes shockingly expensive. For teams using managed observability platforms, trace storage and evaluation can become a significant infrastructure cost, sometimes rivaling model inference itself. A common scenario: a team ships their beautiful instrumentation spec from development into production, and two months later the CFO is asking why the observability line item is bigger than cloud compute. That's not a conversation anyone wants to have. This lesson teaches how to design a sampling strategy (a plan for which traces to keep and which to skip) that preserves the ability to detect failures while keeping costs under control. The goal is to solve for three competing constraints simultaneously: evidence sufficiency, cost limits, and operational reliability. By the end, a production-ready sampling spec that can be handed to an infrastructure team will be complete. A document that justifies itself to the CFO. Not a guess. A calculation.