Prioritization and iteration using evaluation evidence

By Week 6 you have a lot of evidence about what is broken in the AI Data Analyst: a failure taxonomy, a driver analysis, experiment results and a findings-to-actions plan. What you do not have yet is an order. Formatting issues hit 15 percent of queries and barely hurt anyone. Policy violations hit under one percent and can start a compliance escalation. Without a method, the fix order comes down to whatever is easiest or whoever argues loudest.

This lesson scores every item in the backlog on seven dimensions, one to five. Four are about impact: user harm, frequency, business criticality, and confidence in the evidence. Three are about velocity: fixability, time to learn, and reversibility. Every score has to cite the artifact it came from. The example weights count user harm twice and time to learn one and a half times, and you can change them as long as you write down why.

You work through a demo backlog, predict which of two items should go first, and then check your pick against the scores. Two checks sit on top of the ranking. Any item with catastrophic harm gets a critical-failure flag, so a rare failure cannot sink to the bottom. And the top item gets a segment check: is it at least twice as bad for any group of users?

Every item near the top also gets acceptance criteria: which metric moves and by how much, which segment has to see it, and what evidence closes the ticket.

The practice is a ranked backlog: the eight failure modes from the lesson, or 8 to 10 items from your own product, scored with citations, weighted and ranked, at least one tail risk flagged, a segment check on the top item, and acceptance criteria for the top five. The extended version reworks the demo scores by hand with the harm weight doubled and then the time to learn weight doubled, to see which items move.