Experimentation and analytics infrastructure for a production conversational-AI system

Instrumentation, BigQuery + dbt, CUPED for variance reduction, and a build-vs-buy memo that kept us out of a multi-year vendor lock-in.

PM/analyst — designed and built the stack · Production system, real users at scale

Problem

• Existing analytics were a Frankenstein of a SaaS tool, ad-hoc SQL, and a lot of spreadsheet handwork.
• Decisions were slow.

Approach

• A clean event model: a small set of canonical events, defined in a schema document the team treated as a contract.
• BigQuery + dbt marts: sessions, conversations, conversions, cohort tables. Tested. Documented. Owned.
• CUPED for variance reduction — typical effective-sample-size gains of 30–50% on high-value metrics.
• Pre-experiment power calculation became a default step; holdouts on the most consequential changes.

Outcome

• Experiments per quarter went up — because the cost of running one dropped to near-zero.
• The team started catching null results early and stopping experiments that would not have detected the effect they cared about.
• The "is this signal real?" back-and-forth largely went away because the signal arrived with confidence intervals attached.