Product System

Context: A conversational-AI product where every interaction was a structured event stream and every change was potentially an experiment. Existing analytics were a Frankenstein of a SaaS tool, ad-hoc SQL, and a lot of spreadsheet handwork. Decisions were slow.

What I built.

1. A clean event model. A small set of canonical events, defined in a schema document the team treated as a contract. New product surfaces had to map to the existing events or extend them via review. This unlocked everything that followed.
2. The warehouse layer (BigQuery + dbt). Raw events landed in BigQuery; dbt models produced an opinionated set of marts: sessions, conversations, conversions, cohort tables. Tested. Documented. Owned.
3. The experimentation layer. Assignment service, exposure logging, and an analysis package. CUPED for variance reduction on metrics with a sensible pre-period — typical effective-sample-size gains of 30–50% on our high-value metrics, which translated to experiments reaching readout in days not weeks.
4. Power and design discipline. Pre-experiment power calculation became a default step. Sequential testing where appropriate. Holdouts on the most consequential changes.
5. Self-serve dashboards. Looker-style dashboards on top of the dbt marts, with clear ownership and explicit “this is the number” guard rails so the team stopped asking the analyst the same question twice.

The build-vs-buy moment. Mid-build, a vendor pitched a turnkey “experimentation platform.” I wrote the build-vs-buy memo — TCO comparison, lock-in analysis, the specific reasons our event model couldn’t be coerced into their schema, and the year-one and year-three cost trajectories. We built, and the memo’s central prediction — that migration costs, not license costs, would dominate — is the one that aged best.

Outcomes.

• Experiments per quarter went up — not because anyone pushed for more, but because the cost of running one dropped to near-zero.
• The team started catching null results early via power analysis and stopping experiments that wouldn’t have detected the effect they cared about.
• The PM/data-science back-and-forth on “is this signal real?” largely went away because the signal arrived with confidence intervals attached.

What I’d do differently. Spent too long perfecting the schema before users were on the system. Should have shipped a coarser event model on day 1 and refined under load.