Variant A (control)
Everyone who entered the test in the control arm.
How many of them completed the action.
Variant B (treatment)
Test settings
95 means you accept a 1-in-20 chance of calling a difference that is not there.
Use two-tailed unless you committed to a direction before seeing the data.
Verdict
Significant
The difference between 5.00% and 6.00% is larger than chance alone would readily produce, at 95% confidence (p = 0.028295).
Rate A
5.00%
Rate B
6.00%
+20.0% relative
z-score
2.1932
p-value
0.028295
Threshold α = 0.0500
Confidence interval on the difference
0.11% to 1.89%
Percentage points, B minus A. This is the range of true differences consistent with your data — usually more useful for a decision than the yes/no verdict above.
Step-by-step solution for your numbers
The whole test, worked out on your own counts — pooled proportion, standard error, test statistic, p-value and interval.
Step 1: Compute each conversion rate
- Formula
- p̂ = conversions ÷ visitors
- Your numbers
- p̂₁ = 250 ÷ 5,000, p̂₂ = 300 ÷ 5,000
- Result
- p̂₁ = 5.0000%, p̂₂ = 6.0000%
Step 2: Pool the two variants under the null hypothesis
- Formula
- p̄ = (x₁ + x₂) / (n₁ + n₂)
- Your numbers
- p̄ = (250 + 300) ÷ (5,000 + 5,000) = 550 ÷ 10,000
- Result
- p̄ = 0.055000
The null hypothesis says both variants share one true rate, so the test estimates that shared rate from the combined data.
Step 3: Compute the standard error of the difference
- Formula
- SE = √( p̄(1 − p̄) · (1/n₁ + 1/n₂) )
- Your numbers
- SE = √( 0.055000 × 0.945000 × (1/5,000 + 1/5,000) )
- Result
- SE = 0.004560
Step 4: Compute the test statistic
- Formula
- z = (p̂₂ − p̂₁) / SE
- Your numbers
- z = (0.060000 − 0.050000) ÷ 0.004560 = 0.010000 ÷ 0.004560
- Result
- z = 2.1932
z counts how many standard errors separate the two observed rates.
Step 5: Convert z to a two-tailed p-value
- Formula
- p = P(|Z| > |z|) = erfc(|z|/√2)
- Your numbers
- p = erfc(2.1932 ÷ √2)
- Result
- p = 0.028295
The p-value is the probability of seeing a gap at least this large if the two variants were truly identical.
Step 6: Compare the p-value with the significance threshold
- Formula
- reject H₀ when p < α, where α = 1 − confidence
- Your numbers
- 0.028295 < 0.0500
- Result
- Significant at 95% confidence
Step 7: Build the confidence interval for the difference
- Formula
- (p̂₂ − p̂₁) ± z_crit · √( p̂₁(1 − p̂₁)/n₁ + p̂₂(1 − p̂₂)/n₂ )
- Your numbers
- 0.010000 ± 1.9600 × 0.004559
- Result
- 0.11% to 1.89% (percentage points)
The interval uses the unpooled standard error, because pooling is only valid while assuming the rates are equal — the very thing the interval is estimating.
The formula
p̄ = (x₁ + x₂) / (n₁ + n₂)
SE = √( p̄(1 − p̄) · (1/n₁ + 1/n₂) )
z = (p̂₂ − p̂₁) / SE
x is conversions and n is visitors in each variant; p̂ is an observed rate and p̄ the pooled rate assumed under the null hypothesis. The p-value is the probability of a standard normal value at least as extreme as z. The confidence interval on the difference uses the unpooled standard error instead, because it is estimating the very difference the pooled figure assumes away.
Reading the verdict without over-reading it
Significance is a statement about chance, not about value. It answers one question — could a gap this large plausibly have appeared if the two variants were identical? — and stays silent on whether the gap is big enough to be worth anything.
That is why this page shows the confidence interval on the difference next to the verdict, and why the interval usually deserves more of your attention. “Significant” with an interval of +0.1 to +0.3 percentage points is a real but tiny effect. “Not significant” with an interval of −4 to +9 points is not a finding of no difference; it is a finding that the experiment was too small to say anything at all.
Two failure modes cause most bad calls, and neither is visible in the arithmetic. Stopping the moment a result crosses the line inflates false positives badly, because a null experiment checked often enough will eventually cross by luck. And the test assumes visitors were assigned independently and counted once — splitting traffic by day, or letting repeat visits count twice, breaks the assumption no matter how large the sample gets.
Deciding how much traffic to collect before you start is the single most effective way to avoid a stop-early false positive. For sizing a sample against a target precision, use the survey sample size calculator
Methodology and sources
This page implements the large-sample two-proportion z-test as specified in the NIST/SEMATECH e-Handbook of Statistical Methods §7.3.3, using the pooled proportion for the test statistic and the unpooled standard error for the interval on the difference. p-values are computed directly from the complementary error function rather than interpolated from a table, so they stay accurate deep into the tail where a lookup table has simply run out of rows.
The handbook also names the boundary of this method's validity, and the page enforces it: when any expected cell count falls below five, the normal approximation is flagged as unreliable and an exact test is recommended in its place, rather than a confident-looking p-value being printed anyway.
All arithmetic runs in your browser. Nothing you enter is transmitted, stored, or logged.
Primary sources
NIST / SEMATECH
e-Handbook of Statistical Methods §7.3.3 — comparing two proportionsThe federal reference specification for this exact test: the large-sample z statistic using the normal approximation to the binomial, and the exact (Fisher) alternative that applies when samples are small.
NIST / SEMATECH
e-Handbook of Statistical Methods — full handbookThe complete handbook, jointly produced by the National Institute of Standards and Technology, covering hypothesis testing, confidence intervals and the assumptions each method depends on.
NIST / SEMATECH
e-Handbook §1.3.6.7.1 — critical values of the normal distributionThe published critical-value table this page reproduces by direct computation rather than lookup, and the reference used to check the figures it returns.
Links go to the publishing agency, so you always read the current figure rather than a copy of it. This page embeds no agency data of its own.
Common questions
What does the p-value actually tell me?
It is the probability of seeing a difference at least as large as yours if the two variants truly performed identically. A p-value of 0.02 means results this lopsided would turn up about 2% of the time by chance alone. It is not the probability that variant B is better, and it is not the probability that you are wrong — both are common misreadings.
Why does the interval use a different standard error from the test?
The test assumes, for the sake of argument, that both variants share one true rate, so it pools the data to estimate that shared rate. The confidence interval is estimating the difference itself, which means it cannot assume the difference is zero — so it uses the unpooled standard error. Using the pooled figure for both is a common shortcut that produces a slightly wrong interval.
Can I stop the test as soon as it turns significant?
No. Checking repeatedly and stopping at the first significant reading dramatically inflates the false-positive rate — with enough peeking, most null experiments will cross the line at some point. Fix the sample size before you start, or use a method designed for sequential testing.
My result is not significant. Does that mean the variants are the same?
It means this experiment cannot tell them apart. Absence of evidence is not evidence of absence. Look at the confidence interval on the difference: if it spans −5 to +7 points, you have learned very little, whereas an interval of −0.3 to +0.4 points genuinely rules out a large effect.
When should I not use this test?
When the expected count in any cell is below about five, the normal approximation behind the z-test is unreliable and an exact method is appropriate instead. It also assumes each visitor is counted once and assigned independently — repeated visits by the same person, or traffic split by day rather than at random, break that assumption regardless of sample size.
Should I use a one-tailed or two-tailed test?
Two-tailed, unless you committed to a specific direction before collecting data and genuinely would not act on a change in the opposite direction. Picking the direction after seeing which way the result went makes the reported p-value roughly twice as generous as it should be.
Is statistical significance the same as a result worth shipping?
No. With a large enough sample, a difference of a hundredth of a percentage point becomes significant. Significance says the difference is probably real; the confidence interval says how big it might be, and only you can say whether that size justifies the change.
This page does the arithmetic. We check the numbers you put into it.
A test tells you which version won. It cannot tell you whether the market research behind the idea was sound in the first place. A $29 Verified Snapshot answers one business question in a short, fully cited brief — and every claim in it is checked against its own source before it ships. Others check that links work. We check that claims are true.
Get a $29 Verified SnapshotNo subscription, nothing to cancel. A brief that does not pass our verification gate is never auto-delivered.
Other calculators
These calculators are provided for general business analysis and are independent informational research — not investment, financial, legal, or tax advice. The arithmetic is standard and openly documented on this page; the assumptions you enter are yours, and the conclusions drawn from them are too.