“Candidate surges ahead.” “New design lifts sales 30%.” Both kinds of headline rest on the same arithmetic: how much a percentage from a sample can wobble by chance. Here is that arithmetic, applied to six illustrative headlines, with the two traps that catch professionals as often as readers.
These headlines are made up to show typical claims; the checks use the standard formulas from AAPOR, Pew Research Center, NIST and Penn State’s statistics courses.
| Illustrative headline | What the math says | Trap |
|---|---|---|
| “Smith surges ahead, 47% to 44%” (poll of 900, ±3.3) | The margin on a 3-point lead is about ±6.2 points, so the race is a statistical tie. | Margin on the lead |
| “Approval slips to 41% from 43%” (two polls of 1,000) | Each poll is ±3.1; the change between them carries about ±4.3 points. A 2-point drop is noise. | Change between polls |
| “Young voters swing 9 points toward Smith” (about 150 young respondents per poll) | Each subgroup estimate is roughly ±7.8 to ±8.0; the change has a margin near ±11.1. Nine points is inside it. | Subgroups |
| “New checkout lifts sales 30%” (40 vs 52 orders per 1,000 visitors) | z = 1.28, p = 0.20; the 95% interval for the difference runs from -0.64 to +3.04 points. Not shown to work yet. | A/B significance |
| “Teal button wins!” (20 colours tested against the control) | With 20 comparisons at 5%, the chance that at least one “wins” by luck alone is 64%, if none is really better. | Multiple comparisons |
| “Variant B wins after we checked every day” | Checking 20 times and stopping at the first p < 0.05 gives a 25% false-positive rate, not 5%. | Peeking |
A poll’s headline margin is z × √(p(1 − p) ÷ n), about ±3 points for 1,000 to 1,100 interviews at 95% confidence. Pew Research Center uses 1,067 as its ±3-point example, and AAPOR notes that doubling a sample from 1,000 to 2,000 cuts the margin by only about a point. The poll margin of error calculator does the arithmetic for any poll.
But headlines are rarely about one percentage. They are about leads, changes and groups, and each has a bigger margin:
And the margin covers sampling error only. Question wording, who agrees to answer and likely-voter screens add error no formula captures, and AAPOR says a margin of sampling error does not apply to opt-in online polls at all.
An A/B test compares two conversion rates with a two-proportion z-test. With 500 sales from 10,000 visitors on version A and 560 from 10,000 on B, the 12% lift has p = 0.058: close, but not significant at 95%, and the 95% interval for the difference (−0.02 to +1.22 points) still includes zero. The A/B test calculator runs the test, gives the interval and tells you how many visitors you need.
Three things go wrong most often:
Slice any poll or test into enough pieces and something will look dramatic. A national poll split by age, race, region and party produces dozens of subgroup numbers, each with a margin two to three times the headline one. An A/B test split by device, browser and traffic source does the same. Treat a subgroup finding as a lead to check in a new sample, not a result, unless the analysis was planned before the data came in.
A 5% significance level means a 5% false-alarm rate only if you look once, at a sample size you chose in advance. Every extra look is another chance for noise to cross the line. The classic figures for equally spaced looks at a test where nothing is different:
| Looks | Chance of a false winner |
|---|---|
| 1 | 5% |
| 2 | 8% |
| 3 | 11% |
| 5 | 14% |
| 10 | 19% |
| 20 | 25% |
| 50 | 32% |
A dashboard checked daily for seven weeks is about 50 looks: roughly a one-in-three chance of crowning a winner between two identical pages. The fixes are dull but effective: choose the sample size first and analyze once, or use a sequential method built for repeated looks. Polls have a version of this too: a campaign that publicizes only its best internal poll is peeking across polls.
In a typical poll of about 1,000, yes. The margin on the lead itself is close to double the poll’s headline margin, about ±6 points.
Combine the two margins: the margin on a change is about √(m₁² + m₂²). For two polls of 1,000 that is about ±4.4 points for one candidate’s share.
They rest on fewer interviews. A subgroup of 200 in a 1,000-person poll has a margin of about ±6.9 points instead of ±3.1.
The data cannot rule out chance. Look at the confidence interval: if it is wide, the test was too small to tell, which is different from showing no effect.
Repeated checking inflates the false-positive rate: stopping at the first p < 0.05 over 5 looks gives about 14% instead of 5%.