Experiment Analytics

Analytics for experiments: A/B testing, statistical significance, and causal inference. Core for product and growth teams.

Key concepts

• Null hypothesis — No difference between control and treatment
• Statistical significance — p-value, confidence intervals
• Power — Probability of detecting a real effect; sample size
• MDE — Minimum detectable effect for a given sample size

Experiment design

• Randomization — Unit of assignment (user, session, page)
• Sample size — Power analysis, duration, traffic allocation
• Guardrail metrics — Metrics to monitor for unintended harm
• Primacy and novelty — Early effects that may not persist

Analysis

• Two-sample tests — t-test, z-test for means and proportions
• CUPED — Variance reduction for faster, more precise experiments
• Segmentation — Heterogeneous treatment effects by segment
• Multiple comparisons — Bonferroni, FDR when testing many variants

Causal inference

• When A/B isn't possible — Diff-in-diff, synthetic control, propensity scoring
• Instrumental variables — When treatment isn't randomly assigned
• Regression discontinuity — Natural experiments at a threshold

Common analyses

• Experiment results with confidence intervals and p-values
• Power and sample size calculations
• CUPED-adjusted analysis for faster ship decisions
• Meta-analysis across experiments