Conv Eval

Context: A production conversational-AI system, serving real users at scale. The team had vibes about quality but no shared definition of it. Releases shipped on intuition; regressions surfaced only through customer complaints.

The problem. “Is this version better than the last one?” is a deceptively simple question. Aggregate metrics (resolution rate, CSAT) move slowly and confound model changes with content changes, prompt changes, and traffic mix. We needed an eval surface that could detect quality changes at the level of a single prompt revision, in a way humans could trust.

Approach.

1. Taxonomy first. I worked with the team to define a small, opinionated taxonomy of failure modes specific to our agent surface — not generic “hallucination / refusal” but the actual things that broke for our users. Each failure mode had a positive and negative example anchor, a written rubric, and an inter-rater target.
2. Golden set, frozen. A curated set of conversations chosen to span the failure-mode space, with known correct outcomes. Updates to the golden set required a code-review-style review.
3. LLM-as-judge with a human spine. Automated grading against the rubric with a strong model, calibrated against human raters and audited regularly. The judge wasn’t ground truth; the human raters were. The judge was a way to scale the human signal with bounded drift.
4. Wired into the deploy pipeline. Every prompt or model change ran the eval as a gate; deltas were posted to the team’s review channel with examples that flipped.

What changed downstream.

• Conversations about “is this better?” became conversations about which failure mode improved and which got worse. That’s a much sharper conversation.
• The team caught two regressions before they shipped that wouldn’t have shown up in aggregate metrics for weeks.
• The roadmap shifted to address the failure modes that were quantitatively dominant rather than the ones that were emotionally salient.

What I’d do differently. The first taxonomy was too granular — we collapsed about a third of the labels in v2 once we saw which distinctions raters couldn’t reliably make. Start coarser; let the data tell you when to split.

Why this shape. In hindsight, the structure here is the same one safety work needs: regression gates before anything reaches users, human judgment as the ground truth, and automation whose drift is bounded and audited rather than trusted. We built it to catch quality regressions; the machinery generalizes.