Designing an evaluation framework for a production conversational-AI system
How do you measure whether a production conversational-AI system is actually getting better? An eval taxonomy, a labeling protocol, and the downstream changes it forced.
PM — designed the framework, led adoption · Production system, real users at scale
Problem
- The team had vibes about quality but no shared definition of it.
- Releases shipped on intuition; regressions surfaced only through customer complaints.
- Aggregate metrics move slowly and confound model changes with content changes, prompt changes, and traffic mix.
Approach
- Taxonomy first: a small, opinionated taxonomy of failure modes, each with example anchors, a written rubric, and an inter-rater target.
- Golden set, frozen — updates required a code-review-style review.
- LLM-as-judge with a human spine: calibrated against human raters and audited regularly.
- Wired into the deploy pipeline: every prompt or model change ran the eval as a gate.
Outcome
- Conversations about "is this better?" became conversations about which failure mode improved and which got worse.
- Caught two regressions before they shipped that would not have shown up in aggregate metrics for weeks.
- The roadmap shifted to the failure modes that were quantitatively dominant rather than the ones that were emotionally salient.
Context: A production conversational-AI system, serving real users at scale. The team had vibes about quality but no shared definition of it. Releases shipped on intuition; regressions surfaced only through customer complaints.
The problem. “Is this version better than the last one?” is a deceptively simple question. Aggregate metrics (task-completion rate, user-satisfaction scores) move slowly and confound model changes with content changes, prompt changes, and traffic mix. We needed an eval surface that could detect quality changes at the level of a single prompt revision, in a way humans could trust.
Approach.
- Taxonomy first. I worked with the team to define a small, opinionated taxonomy of failure modes specific to our agent surface — not generic “hallucination / refusal” but the actual things that broke for our users. Each failure mode had a positive and negative example anchor, a written rubric, and an inter-rater target.
- Golden set, frozen. A curated set of conversations chosen to span the failure-mode space, with known correct outcomes. Updates to the golden set required a code-review-style review.
- LLM-as-judge with a human spine. Automated grading against the rubric with a strong model, calibrated against human raters and audited regularly. The judge wasn’t ground truth; the human raters were. The judge was a way to scale the human signal with bounded drift.
- Wired into the deploy pipeline. Every prompt or model change ran the eval as a gate; deltas were posted to the team’s review channel with examples that flipped.
What changed downstream.
- Conversations about “is this better?” became conversations about which failure mode improved and which got worse. That’s a much sharper conversation.
- The team caught two regressions before they shipped that wouldn’t have shown up in aggregate metrics for weeks.
- The roadmap shifted to address the failure modes that were quantitatively dominant rather than the ones that were emotionally salient.
What I’d do differently. The first taxonomy was too granular — we collapsed about a third of the labels in v2 once we saw which distinctions raters couldn’t reliably make. Start coarser; let the data tell you when to split.
Why this shape. In hindsight, the structure here is the same one safety work needs: regression gates before anything reaches users, human judgment as the ground truth, and automation whose drift is bounded and audited rather than trusted. We built it to catch quality regressions; the machinery generalizes.