Reading the verdict without over-reading it
Significance is a statement about chance, not about value. It answers one question — could a gap this large plausibly have appeared if the two variants were identical? — and stays silent on whether the gap is big enough to be worth anything.
That is why this page shows the confidence interval on the difference next to the verdict, and why the interval usually deserves more of your attention. “Significant” with an interval of +0.1 to +0.3 percentage points is a real but tiny effect. “Not significant” with an interval of −4 to +9 points is not a finding of no difference; it is a finding that the experiment was too small to say anything at all.
Two failure modes cause most bad calls, and neither is visible in the arithmetic. Stopping the moment a result crosses the line inflates false positives badly, because a null experiment checked often enough will eventually cross by luck. And the test assumes visitors were assigned independently and counted once — splitting traffic by day, or letting repeat visits count twice, breaks the assumption no matter how large the sample gets.
Deciding how much traffic to collect before you start is the single most effective way to avoid a stop-early false positive. For sizing a sample against a target precision, use the survey sample size calculator
Methodology and sources
This page implements the large-sample two-proportion z-test as specified in the NIST/SEMATECH e-Handbook of Statistical Methods §7.3.3, using the pooled proportion for the test statistic and the unpooled standard error for the interval on the difference. p-values are computed directly from the complementary error function rather than interpolated from a table, so they stay accurate deep into the tail where a lookup table has simply run out of rows.
The handbook also names the boundary of this method's validity, and the page enforces it: when any expected cell count falls below five, the normal approximation is flagged as unreliable and an exact test is recommended in its place, rather than a confident-looking p-value being printed anyway.
All arithmetic runs in your browser. Nothing you enter is transmitted, stored, or logged.
Primary sources
NIST / SEMATECH
e-Handbook of Statistical Methods §7.3.3 — comparing two proportionsThe federal reference specification for this exact test: the large-sample z statistic using the normal approximation to the binomial, and the exact (Fisher) alternative that applies when samples are small.
NIST / SEMATECH
e-Handbook of Statistical Methods — full handbookThe complete handbook, jointly produced by the National Institute of Standards and Technology, covering hypothesis testing, confidence intervals and the assumptions each method depends on.
NIST / SEMATECH
e-Handbook §1.3.6.7.1 — critical values of the normal distributionThe published critical-value table this page reproduces by direct computation rather than lookup, and the reference used to check the figures it returns.
Links go to the publishing agency, so you always read the current figure rather than a copy of it. This page embeds no agency data of its own.
Common questions
What does the p-value actually tell me?
It is the probability of seeing a difference at least as large as yours if the two variants truly performed identically. A p-value of 0.02 means results this lopsided would turn up about 2% of the time by chance alone. It is not the probability that variant B is better, and it is not the probability that you are wrong — both are common misreadings.
Why does the interval use a different standard error from the test?
The test assumes, for the sake of argument, that both variants share one true rate, so it pools the data to estimate that shared rate. The confidence interval is estimating the difference itself, which means it cannot assume the difference is zero — so it uses the unpooled standard error. Using the pooled figure for both is a common shortcut that produces a slightly wrong interval.
Can I stop the test as soon as it turns significant?
No. Checking repeatedly and stopping at the first significant reading dramatically inflates the false-positive rate — with enough peeking, most null experiments will cross the line at some point. Fix the sample size before you start, or use a method designed for sequential testing.
My result is not significant. Does that mean the variants are the same?
It means this experiment cannot tell them apart. Absence of evidence is not evidence of absence. Look at the confidence interval on the difference: if it spans −5 to +7 points, you have learned very little, whereas an interval of −0.3 to +0.4 points genuinely rules out a large effect.
When should I not use this test?
When the expected count in any cell is below about five, the normal approximation behind the z-test is unreliable and an exact method is appropriate instead. It also assumes each visitor is counted once and assigned independently — repeated visits by the same person, or traffic split by day rather than at random, break that assumption regardless of sample size.
Should I use a one-tailed or two-tailed test?
Two-tailed, unless you committed to a specific direction before collecting data and genuinely would not act on a change in the opposite direction. Picking the direction after seeing which way the result went makes the reported p-value roughly twice as generous as it should be.
Is statistical significance the same as a result worth shipping?
No. With a large enough sample, a difference of a hundredth of a percentage point becomes significant. Significance says the difference is probably real; the confidence interval says how big it might be, and only you can say whether that size justifies the change.
This page does the arithmetic. We check the numbers you put into it.
A test tells you which version won. It cannot tell you whether the market research behind the idea was sound in the first place. A $29 Verified Snapshot answers one business question in a short, fully cited brief — and every claim in it is checked against its own source before it ships. Others check that links work. We check that claims are true.
Get a $29 Verified SnapshotNo recurring fee, nothing to cancel. A brief that does not pass our verification gate is never auto-delivered.
Other calculators
These calculators are provided for general business analysis and are independent informational research — not investment, financial, legal, or tax advice. The arithmetic is standard and openly documented on this page; the assumptions you enter are yours, and the conclusions drawn from them are too.