Evaluation cadence

In Lesson 6.1, you made a ship/ramp/hold decision for the v2 change — who in your organization has the authority to override that recommendation, and what evidence threshold would justify an override?

Six months post-ship, judge drift goes undetected — nobody owns calibration, nobody checks Kappa, production incident forces emergency remediation

Three-tier framework balances thoroughness (catching real issues) with practicality (not drowning teams in meetings) — weekly checks, monthly deep dives, quarterly audits

Weekly quality check (30-45 min) — automated metric review, dashboard review, incident triage — flags issues for deeper analysis or immediate rollback

• Automated metric review (15 min) — latency, cost, error rates, blocking metrics (must-pass thresholds for deploys)
• Dashboard review (20 min) — any metrics outside acceptable ranges?
• Incident triage (10 min) — new failure modes or user reports requiring investigation

Monthly deep dive (90 min) — metric trend analysis, judge re-calibration check, segment health review, evaluation debt review — updates thresholds or escalates risks

Quarterly regression audit (half day) — regression suite refresh, full judge re-calibration, experiment backlog prioritization, evaluation system health — major updates or roadmap adjustments

RACI matrix — every evaluation activity has exactly one Accountable person and at least one Responsible person — blank Accountable cells mean nobody owns the decision

Evaluation debt register makes invisible risk visible — tracks deferred work with risk level, target paydown date, and actual paydown date

Cadence calendar shows 6.2 hours/month for a 5-person team — well under 10% of team capacity — weekly checks, monthly deep dives, and quarterly audits are operationally sustainable

Monthly deep dive would have caught judge drift in Month 2 — Kappa 0.64 triggers 'Full re-calibration needed' debt entry — quarterly audit fixes it before 6 months of silent failures

Build a three-document Evaluation Cadence Package — cadence calendar, RACI matrix, evaluation debt register — operational governance that keeps evaluation from becoming invisible work

Skipping Tier 2 (monthly deep dive) accumulates silent risk — chronic degradation invisible to weekly checks and quarterly audits too late — judge drift example shows 12 weeks of incorrect SQL approval

Apply cadence reasoning to judge drift, RACI conflicts, and evaluation debt risk translation — knowledge check scenarios

Three-tier pyramid — weekly checks (52×/year, acute issues), monthly deep dives (12×/year, chronic degradation), quarterly audits (4×/year, strategic alignment)

RACI ownership assigns one Accountable person per tier — debt register tracks deferred work with risk levels