Rigorously
A practice environment for designing rigorous online experiments — power, CUPED variance reduction, common pitfalls.
Most “A/B testing tutorials” stop at “alpha = 0.05, sample size formula, done.” That’s where the actual problems start. Rigorously is the tool I wished existed when I was learning experimentation properly: a step-through coach that drills the design choices that matter.
It walks through framing the metric, choosing a pre-period for CUPED, doing power analysis honestly (including the moment where you realize the experiment can’t detect the effect you care about), and the diagnostic moves when your readout looks suspicious. It’s opinionated, which is the point — most experimentation goes wrong because the design was sloppy, not because the analysis was.
The habits it drills come from running real experiments on a production conversational-AI system, where sloppy design costs weeks of traffic. A rehosted public version is in the works.