Two-proportion z-test for conversion rates, the confidence interval for the lift, the sample size you need, and what peeking does to your error rate
Updated October 2026
This is the large-sample test for two proportions in the NIST/SEMATECH e-Handbook of Statistical Methods. The calculator reproduces Penn State’s STAT 500 example (52 of 69 vs 120 of 131: z = 3.15, p = 0.002, 95% interval for the difference 0.050 to 0.275). For very small samples, use Fisher’s exact test instead.
The confidence interval for the difference tells you what lift is plausible. In the default example it runs from slightly negative to +1.22 points, so B could be a little worse than A or as much as about 24% better. “Not significant” here means the test was too small to tell, not that B has no effect. The relative lift interval is the point interval divided by A’s rate, a common approximation.
| Test | Lift | 95% CI for difference (points) | z | p-value | Significant at 95%? |
|---|---|---|---|---|---|
| Default: 500 vs 560 of 10,000 each | +12.0% | -0.02 to +1.22 | 1.89 | 0.0583 | No |
| Same rates, twice the traffic | +12.0% | +0.16 to +1.04 | 2.68 | 0.0074 | Yes |
| Small test: 40 vs 55 of 1,000 | +37.5% | -0.36 to +3.36 | 1.58 | 0.1148 | No |
| Big site: 2.00% vs 2.10% of 200,000 | +5.0% | +0.01 to +0.19 | 2.23 | 0.0256 | Yes |
| PSU textbook: 52 of 69 vs 120 of 131 | +21.6% | +5.02 to +27.46 | 3.15 | 0.0017 | Yes |
The first two rows have identical conversion rates; doubling the traffic turns a borderline result into a clear one. The big-site row shows the opposite problem: with enough traffic a 0.1-point difference is “significant” even though it may not be worth acting on.
The visitors needed per group to detect a given relative lift at 95% confidence (two-sided), using the formula in Penn State’s STAT 509 course: n = 2(zα + zβ)² p̄(1 − p̄) ÷ (p₁ − p₂)². The same formula gives that course’s example of 79 per group for 25% vs 50% at 90% power.
| Baseline conversion | Relative lift to detect | Per group, 80% power | Per group, 90% power |
|---|---|---|---|
| 2% | 5% | 315,208 | 421,973 |
| 2% | 10% | 80,683 | 108,012 |
| 5% | 5% | 122,125 | 163,491 |
| 5% | 10% | 31,235 | 41,815 |
| 5% | 20% | 8,159 | 10,923 |
| 10% | 10% | 14,752 | 19,749 |
| 20% | 10% | 6,511 | 8,716 |
Halving the lift you want to detect roughly quadruples the sample, and low baseline rates need far more traffic. A 5% lift on a 2% conversion rate needs hundreds of thousands of visitors per group.
A 5% significance level assumes you look once, at a sample size fixed in advance. If you check results as they come in and stop the first time p dips below 0.05, the chance of declaring a winner when there is no real difference climbs with every look. These are the classic figures for equally spaced looks (Armitage, McPherson and Rowe, tabulated by Pocock in 1977), which our own 400,000-run simulation reproduced:
| Number of looks | Chance of a false “winner” |
|---|---|
| 1 | 5% |
| 2 | 8% |
| 3 | 11% |
| 5 | 14% |
| 10 | 19% |
| 20 | 25% |
| 50 | 32% |
Checking daily for seven weeks is about 50 looks, so a test of two identical pages would “win” roughly a third of the time. Fix the sample size first and analyze once, or use a sequential method designed for repeated looks (group sequential boundaries or always-valid p-values) that keeps the error rate at its stated level.
It handles one control and one variant with a yes/no outcome. Revenue per visitor or time on page need a t-test on means; three or more variants need a chi-square test or a correction for multiple comparisons; sequential and Bayesian testing need their own tools.