Trace design and reproducibility

A PM asks why a user got a conversion rate of 4.2% when the dashboard says 3.8%. You open the trace and it says success true, latency 2,300 milliseconds. Was the SQL wrong, did retrieval pull the wrong metric definition, or did the summary misread the number? A trace that only records that the system ran can’t tell you.

A trace is the record of one request’s path through the system. It is split into spans, one per stage: retrieval, SQL, chart and narrative. Each span holds the fields that stage needs, because each stage fails in its own way. Retrieval gets the most fields, since a retrieval failure doesn’t throw an error. It just produces a confident wrong answer.

Traces are stored as nested JSON and flattened into one row per request for analysis, using spans_to_row. From there you can measure completeness, the share of schema fields that actually have data. Against a 20-field schema, a v0 trace fills 6 and a v1 trace fills 19.

You predict which of two retrieval schemas lets you prove retrieval caused a wrong DAU definition, then compare a v0 and a v1 trace for the same request. You also see what it takes to replay a request after a fix: the version fields from 2.1 plus what each span took in and put out.

The lesson closes on three mistakes: logging everything, changing the schema without a schema_version field, and deploying logging nobody tested.

The practice is an Instrumentation Readiness Report built from the v0 and v1 traces in the lesson: completeness for each, a schema comparison table where every field names the measurement it enables, and a chart rendering span you design yourself. The extended version writes the rules a checker would apply to traces against your schema.