How instrumentation requirements differ by system type

A v1 span can tell you that SQL generation failed. It can’t tell you whether the system picked the wrong tool, wrote bad SQL, or had its query rejected by the database. Those three causes need different fixes, and they all show up as the same status field.

What you need to log depends on the kind of system. An LLM app needs the prompt template version, the model config, a schema validation result, token counts and finish_reason, so you can tell a truncated answer from a wrong one. A RAG system needs every step between the question and the context the model saw: the search query, the candidate documents with their scores and rank, any reranking, the chunks passed to the generator and the citations. An agent needs each stage of a tool call logged separately (which tool it chose, the arguments it wrote, whether the call ran, and what it did with the result) plus the transitions between states.

The AI Data Analyst uses all three patterns. Query parsing, charts and the narrative are LLM apps, context retrieval is RAG, and SQL generation and execution are tool calls. You predict which stage has the biggest gap in v1, then check it. In SQL generation only execution is fully logged. The decisions before it, which tool and which arguments, are missing.

You also pull the state transitions out of the traces and look for paths that were never designed. In v1, 124 traces go from running the SQL straight to the narrative with no chart step. You write hypotheses for that path and name the field that would settle it.

The practice is a Delta Instrumentation Spec built from the lesson’s v1 findings: each stage classified by system type, the evaluation questions you can and can’t answer today, three hotspots from the transition counts, and a table of missing fields with the evaluation each one unblocks.