Designing an evaluation framework for a production conversational-AI system
How do you measure whether a production conversational-AI system is actually getting better? An eval taxonomy, a labeling protocol, and the downstream changes it forced.
PM — designed the framework, led adoption · Production system, real users at scale
LLM-as-judgeLabeling protocolGolden setFailure-mode taxonomyHuman-rater calibrationDeploy-pipeline gate
- Problem
-
- The team had vibes about quality but no shared definition of it.
- Releases shipped on intuition; regressions surfaced only through customer complaints.
- Aggregate metrics move slowly and confound model changes with content changes, prompt changes, and traffic mix.
- Approach
-
- Taxonomy first: a small, opinionated taxonomy of failure modes, each with example anchors, a written rubric, and an inter-rater target.
- Golden set, frozen — updates required a code-review-style review.
- LLM-as-judge with a human spine: calibrated against human raters and audited regularly.
- Wired into the deploy pipeline: every prompt or model change ran the eval as a gate.
- Outcome
-
- Conversations about "is this better?" became conversations about which failure mode improved and which got worse.
- Caught two regressions before they shipped that would not have shown up in aggregate metrics for weeks.
- The roadmap shifted to the failure modes that were quantitatively dominant rather than the ones that were emotionally salient.