Designing experiments that establish cause, not just correlation
Most analysis is observational: you look at what happened and find patterns. A/B testing is different — it lets you deliberately make something happen to a random subset of users and measure the causal effect. That distinction matters enormously. The fact that users who engage with Feature X retain better doesn't mean Feature X causes retention — both could be driven by a third factor like user intent. A properly randomised experiment eliminates that confound and establishes causation directly.
This notebook builds the complete picture: the statistical machinery from first principles, the practical engineering of experiments at scale, and the structured reasoning needed to walk through an A/B testing case in an interview. It draws on the hypothesis testing framework from Statistics 8, confidence intervals from Statistics 6, and goodness-of-fit tests from Statistics 9.
A/B testing works when you can randomise users into groups, measure an outcome within a reasonable timeframe, and the treatment affects only the unit it's assigned to. Common use cases: UI changes, recommendation or ranking algorithm changes, notification copy, onboarding flows, pricing.
It gives misleading results in several situations worth knowing cold:
Always flag change aversion and novelty effect as threats to validity. A significant result in week one can't always be trusted — you need sufficient duration for these effects to wash out before drawing conclusions.
Every A/B test follows the same logical arc, whether at a startup or at Google. Walk through each step in order — skipping ahead is how experiments go wrong.
Before touching statistics you need to understand what success looks like. This means mapping the user funnel, identifying where the change intervenes, and translating business goals into measurable quantities.
A user funnel is the sequence of actions from first contact to the desired outcome. For a search-and-purchase product: visit → search → browse → click → purchase. The funnel matters for two reasons: it tells you where the treatment kicks in — defining which users should even be enrolled in the experiment — and it surfaces the right metric to measure at each stage. Don't enroll users who were never exposed to the change; they dilute your effect size and inflate the sample size you need.
Every experiment needs three types of metrics defined upfront:
| Type | Description | Example |
|---|---|---|
| Success metric | The primary north-star. Should be simple, stable, and causally connected to the treatment. Rarely changes between experiments. | Revenue per user per day |
| Driver metric | Short-term, faster-moving proxy for the success metric. Sensitive to the specific change. Actionable — you can directly engineer it. | Click-through rate on search results |
| Guardrail metric | Must not degrade. Protects against winning on one dimension while breaking something important elsewhere. | Page load time, error rate, unsubscribe rate |
When multiple metrics compete, a single Overall Evaluation Criterion (OEC) forces trade-offs to be explicit: normalise each metric to a fixed range, assign weights, and sum. \(\text{OEC} = \sum_i w_i \cdot \tilde{m}_i.\) The weights should be agreed with stakeholders before the experiment runs, not chosen after you see which version of the weights makes the result look good.
With a metric chosen, formalise what you're testing. The statistical machinery is the two-sample t-test from Statistics 8.
\(H_0: \mu_T = \mu_C\) — treatment has no effect on the mean metric
\(H_1: \mu_T \neq \mu_C\) — treatment shifts the mean metric (two-sided)
Two parameters bound the test before it starts:
| H₀ true (no effect) | H₁ true (real effect) | |
|---|---|---|
| Reject H₀ | Type I error — false positive (rate = α) | Correct — true positive (rate = power) |
| Fail to reject H₀ | Correct — true negative | Type II error — false negative (rate = β) |
The randomisation unit is what gets assigned to control or treatment. Choosing it correctly has both practical and statistical consequences.
| Unit | Pros | Cons |
|---|---|---|
| User ID | Stable across sessions and devices; enables long-run measurement (retention, LTV) | Requires login; identifiable |
| Cookie | Anonymous; works for logged-out users | Lost if user clears browser; not portable across devices |
| Session | Higher granularity → more units → more power for short-horizon metrics | Same user can see both variants; inconsistent experience |
| Device ID | Immutable per device; stable | Mobile only; identifiable |
| IP address | Sometimes the only option for infrastructure experiments (e.g. latency) | Shared IPs common; changes with location |
There is one hard constraint: the randomisation unit must be the same as, or coarser than, the unit of analysis. Here's why.
The variance formula \(\text{Var}(\bar X) = \sigma^2/n\) assumes the \(n\) observations are independent. If you randomise by user but count page views, page views from the same user are correlated — they share treatment assignment. Treating correlated observations as independent deflates the estimated standard error, makes confidence intervals too narrow, and inflates the Type I error rate above your chosen \(\alpha\). The effective sample size is somewhere between \(n_{\text{views}}\) and \(n_{\text{users}}\), not \(n_{\text{views}}\). User-level randomisation is the default precisely because it guarantees independence at the most common unit of analysis.
The randomisation unit vs. unit of analysis mismatch is the most common source of inflated false positive rates in practice. It matters that you understand why the rule exists — not just that randomisation unit must be ≥ unit of analysis, but that violating it breaks the independence assumption the variance formula depends on.
Only enroll users at the point where they actually encounter the treatment. If you are testing a new search ranking algorithm, users should enter the experiment when they submit a search query — not when they land on the homepage. Enrolling users who never hit the treated surface dilutes the measured effect and forces you to collect far more data than necessary.
One caution: be careful if your enrollment criterion is itself affected by the treatment. If you filter on "active users" and the treatment changes what counts as active, you've introduced selection bias that makes control and treatment groups incomparable before the treatment even starts. This violates the Stable Unit Treatment Value Assumption (SUTVA): the outcome for any unit should depend only on that unit's treatment assignment, not on other units' assignments or on the measurement process itself.
The required sample size per variant follows from inverting the power condition. We want \(\mathbb{P}[\text{reject } H_0 \mid H_1 \text{ true}] = 1 - \beta.\) Working through the two-sample z-test:
\[n = \frac{(\sigma_T^2 + \sigma_C^2)(z_{1-\alpha/2} + z_{1-\beta})^2}{\delta^2}\]For the standard choices \(\alpha = 0.05\) and \(1 - \beta = 0.80\): \(z_{0.975} \approx 1.96\), \(z_{0.80} \approx 0.84\), so \((1.96 + 0.84)^2 \approx 7.84.\) With equal-sized groups (\(\sigma_T^2 \approx \sigma_C^2 = \sigma^2\)) this gives the rule of thumb you should memorise:
\[n \approx \frac{16\sigma^2}{\delta^2}\]where \(\sigma^2\) is the metric variance (estimated from historical data or an A/A test) and \(\delta\) is the minimum detectable effect (MDE) — the smallest change that would be meaningful in practice. For binary metrics like click-through rate, the maximum variance is 0.25 (when the base rate is 0.5).
If the sample size is too large to collect in a reasonable time, you have three levers: relax the MDE (test only for larger effects), reduce metric variance (use a more targeted metric or cap outliers), or reduce power (accept a higher miss rate).
Two constraints on duration:
The one-week minimum is easy to forget when sample size calculations suggest you can finish in three days. The formula gives you statistical sample size; the day-of-week effect gives you the minimum calendar duration. Novelty and primacy effects are additional reasons to run longer — short experiments capture atypical behaviour.
Before interpreting any number, confirm the experiment ran as designed. A flawed experiment produces results that should be ignored regardless of how significant they look. This step comes before the statistical interpretation — not after.
With 50/50 randomisation you expect approximately equal group sizes. A large discrepancy signals a bug in the assignment pipeline, a logging issue, or a differential dropout rate between variants. Test with a chi-squared goodness-of-fit test — the same machinery as Statistics 9:
\[\chi^2 = \sum_{i \in \{C,T\}} \frac{(O_i - E_i)^2}{E_i}, \quad E_i = \frac{N}{2}\]Under the null of fair assignment this follows \(\chi^2_1\). A significant result (p < 0.05) is a red flag — stop and debug before proceeding. Do not try to correct for an SRM post-hoc; the bias in which users ended up in each group cannot be removed by reweighting.
Before running the real experiment, run a pre-period where both groups receive identical treatment (both see control). This validates that your instrumentation is unbiased, that the randomisation is working correctly, and that the groups are statistically homogeneous before any treatment is applied. If an A/A test finds a significant difference, something is broken upstream — investigate before trusting any A/B results.
Audit whether major external events fell within the experiment window: competitor launches, marketing campaigns, public holidays, economic disruptions. If a confounding event is coincident with the treatment rollout, the effects are inseparable. Rerun the experiment in a cleaner window rather than try to estimate and subtract the external effect.
Segment results by new users vs. returning users. Novelty effect shows up as a large treatment effect for returning users that decays over time — they're engaging because something changed, not because it's better. Primacy effect is the opposite: returning users resist change, suppressing the true treatment effect. A genuine improvement should show a consistent signal across both segments and across time within the experiment window.
The p-value is the probability, under \(H_0\), of observing a test statistic at least as extreme as the one you computed. It is emphatically not the probability that \(H_1\) is true. If \(p < \alpha\), reject \(H_0\) and conclude the effect is statistically significant. See Statistics 8 for the full mechanics of the two-sample t-test and Wald's test.
Statistical significance means the observed effect is unlikely under the null. Practical significance means the effect is large enough to matter in the real world. With very large samples you can detect effects that are statistically significant but economically negligible. Always evaluate the confidence interval against the MDE \(\delta_{\min}\), not just against zero.
The key output is the 95% confidence interval for the lift \(\hat\delta = \bar X_T - \bar X_C\), read against two reference lines: zero and the practical significance boundary \(\delta_{\min}\).
Testing \(N\) metrics simultaneously each at level \(\alpha = 0.05\) inflates the family-wise error rate:
\[\mathbb{P}[\text{at least one false positive}] = 1 - (1 - 0.05)^N\]For \(N = 10\) this is 40%; for \(N = 20\) it's 64%. Three responses, in order of conservatism:
A trend visible in aggregate data can reverse in every subgroup when a confounding variable is correlated with both group assignment and the outcome. This is why post-hoc segmentation is dangerous: segment across 20 dimensions and you'll find at least one slice that shows the opposite result. Pre-register your subgroup analyses before the experiment runs — decide which segments you care about based on the hypothesis, not based on what the data shows after the fact.
Stopping an experiment as soon as \(p < 0.05\) inflates the Type I error rate far above 5%. The reason is sequential testing: each time you peek at the data and run a significance test, you're making a new implicit decision. Across 20 peeks all under \(H_0\), the probability that at least one peek yields \(p < 0.05\) is close to 64% — even with no real effect. Commit to your sample size and duration before the experiment starts, and do not stop early based on the p-value.
If you genuinely need interim analyses (for safety monitoring or resource constraints), use sequential testing methods (e.g. alpha spending functions) that correct for multiple looks while preserving the overall Type I error rate.
On social or marketplace platforms, treatment and control users interact. If a better search ranking in the treatment arm converts more buyers, it depletes inventory for control users — creating a negative spillover that makes the treatment look even better relative to control. The effect you measure is not the effect you'd see at full rollout. Mitigations:
Slicing results across many dimensions after the fact is equivalent to running many parallel hypothesis tests. Each additional segment multiplies the chance of a false positive. If subgroup analysis is scientifically important, pre-register the specific segments with clear hypotheses about direction and magnitude before collecting data.
Statistical significance is necessary but not sufficient to ship. Three additional factors always enter the decision:
When you do ship, always ramp gradually rather than flipping to 100% immediately. Real-world behaviour at full traffic differs from 10% traffic in ways that are hard to predict: event-driven seasonality, interaction effects with concurrent experiments, and infrastructure load can all shift the metrics. Monitor guardrail metrics closely during ramp-up and be ready to roll back.
When randomisation is impossible — historical data, ethical constraints, marketplace interference — causal inference methods offer structured alternatives. All require assumptions that A/B testing avoids by design; the key is knowing which assumption you're making and whether it's defensible.