Retrieval metrics: precision, recall, MRR and NDCG

A user asks the AI Data Analyst for mobile checkout conversion in Q4. Retrieval, the step that looks up schema context before any SQL is written, comes back with documents about desktop conversion. The SQL runs, the summary describes it accurately, and the numbers are for desktop users. SQL execution success is 0.91 and summary faithfulness is 0.78, so nothing you are measuring points at the problem. That is why retrieval gets evaluated on its own.

Similarity scores like cosine similarity tell you how close a query and a document are. A mobile question and a desktop document can be very close and still wrong for each other. Retrieval metrics compare the ranked list against labels a person assigned. Precision@k measures noise in the top k. Recall@k measures how many of the relevant documents came back at all. MRR looks at how high the first relevant document sits. NDCG uses graded labels and rewards putting the most relevant documents first.

You compute each one by hand for a single query, see why precision and recall can match in one example and split apart as soon as k changes, and learn to pick the metric from what the product does with the documents: recall for legal search, MRR for a one-answer bot, precision for a small context window, NDCG for graded, ordered results.

The practice is a small Retrieval Quality Report on the two example queries from the lesson: mean precision and recall at 3 and at 5 and MRR, a read on the precision and recall tradeoff, and a recommendation. The extended version works an adversarial drop as points and as a percentage, and traces one retrieval miss to the wrong answer it caused.