You ran an experiment. The new button converted at 11.2% and the old one at 9.6%, so you are ready to declare a winner — but is that gap real, or just the natural wobble of small numbers? This A/B test significance calculator runs a two-proportion z-test on your control and variant results and tells you, in plain terms, whether the difference you are seeing is statistically significant or could easily have happened by chance.
Enter the conversions and visitors for each arm, pick your confidence level (95% by default), and the result card shows the conversion rates, the z-score, the two-sided p-value, the relative lift, and a clear Significant / Not Significant verdict. No statistics degree required.
How to Use
- Enter the Control (A) Conversions — how many visitors completed your goal (a purchase, signup, click, download) in the original version.
- Enter the Control (A) Visitors — the total number of people who saw the control arm.
- Enter the Variant (B) Conversions — completions in the new version you are testing.
- Enter the Variant (B) Visitors — total people who saw the variant.
- Set the Confidence Level. 95% is the standard business threshold; choose 99% only when a false positive would be costly, because it needs a much larger sample.
- Read the result card. If the verdict is Significant, the observed lift is unlikely to be luck; if it is Not Significant, keep collecting data or kill the test — the current evidence is too weak to decide.
The engine computes both conversion rates (conversions divided by visitors), then pools them into a single expected rate under the null hypothesis that A and B are identical. It measures how far apart the two rates are in standard-error units (the z-score), and converts that into a p-value — the probability of seeing a gap this large if there were truly no difference at all.
The Math Behind the Verdict
This calculator uses the pooled two-proportion z-test, the same method behind statsmodels' proportions_ztest and most enterprise experimentation platforms:
z = (pB − pA) / √(p_pooled × (1 − p_pooled) × (1/nA + 1/nB))
Where pA and pB are the conversion rates, nA and nB are the sample sizes, and p_pooled is the combined conversion rate across both arms. A two-sided p-value is then read from the standard normal distribution. When the p-value drops below your threshold (0.05 at 95% confidence, |z| > 1.96), you reject the idea that the two arms are the same and call the result significant.
Worked example: 480 conversions from 5,000 control visitors (9.6%) versus 560 from 5,000 variant visitors (11.2%). The pooled rate is 10.4%, the standard error works out to about 0.61 percentage points, and the z-score is roughly 2.62. That gives a two-sided p-value near 0.009 — well under 0.05 — so the 16.7% relative lift is statistically significant at the 95% level.
How to Read the Result
- P-Value — the probability of a difference this extreme if B were really no better than A. Below 0.05 = significant; above 0.05 = inconclusive.
- Z-Score — how many standard errors separate the two rates. Beyond ±1.96 is the 95% line; beyond ±2.58 is 99%.
- Relative Lift — the percentage change in conversion rate from A to B. A big lift that is not significant is still noise until the sample is large enough.
- Significant / Not Significant — the bottom-line call at your chosen confidence level.
Examples
Example 1 — A landing page redesign. Control A gets 480 signups out of 5,000 visitors (9.6%); variant B gets 560 out of 5,000 (11.2%). The calculator reports a z-score of about 2.62, a p-value near 0.009, and a 16.7% relative lift. At 95% confidence this is a significant win — roll out the redesign.
Example 2 — A checkout button color. Old color: 120 purchases out of 2,000 visitors (6.0%). New color: 132 out of 2,000 (6.6%). The raw lift is 10%, but with such small counts the z-score lands near 0.7, the p-value above 0.4, and the verdict is Not Significant. Do not ship a color change based on this; the difference is indistinguishable from random variation.
Example 3 — A high-traffic experiment. Variant B reaches 55,000 conversions from 2,000,000 visitors versus 50,000 from 2,000,000 in control — a lift from 2.5% to 2.75%. At this volume even a modest 10% relative gap produces a very large z-score and a p-value near zero, so it is overwhelmingly significant. Larger samples detect smaller true effects.
FAQ
What does "statistically significant" actually mean?
It means the difference between your control and variant is unlikely to have occurred by random chance alone, assuming the two versions were truly identical. At 95% confidence, a p-value below 0.05 tells you there is less than a 5% probability of seeing a gap this large if B were no better than A. It does not prove the lift is large or important — it only rules out luck as the explanation.
What is a good p-value?
Below 0.05 at the 95% level is the standard threshold for business experiments. Below 0.01 (99% confidence) is stronger evidence. A p-value between 0.05 and 1.0 is inconclusive: you have not proven a winner, and you should not ship the change yet. Never choose a threshold after seeing the result — pick 95% before the test starts.
How much traffic do I need?
There is no single number; it depends on your baseline conversion rate and how small a lift you want to detect. As a rule of thumb, detecting a 10% relative lift on a 5% baseline typically requires tens of thousands of visitors per arm. This calculator tells you whether your current sample is already decisive; for sample-size planning before a test, use a dedicated sample-size calculator and aim for at least thousands per group.
Why use a z-test instead of just comparing percentages?
Comparing 9.6% to 11.2% ignores the uncertainty around each rate. A z-test accounts for sample size: the same 1.6-point gap means something very different on 100 visitors versus 1,000,000. The test asks whether the gap is large relative to the noise in your data, which is exactly the question that matters.
Can a result be significant but still a bad business decision?
Yes. Significance only says the effect is real; it does not say it is profitable. A statistically significant 0.2% lift that costs a large engineering investment, or a significant drop-off in another metric, may still be the wrong call. Always weigh statistical significance against effect size, cost, and downstream metrics like retention or revenue.
What if one group has zero conversions?
That is a valid, though extreme, input. If the control has zero conversions and the variant does not, the z-score becomes very large and the test can show significance — but with zero conversions your estimate is unstable. Treat such results cautiously and gather more data before making a permanent decision.
Disclaimer
This A/B test significance calculator is provided for informational purposes only and does not constitute statistical, business, or legal advice. Results are estimates based on the two-proportion z-test and the inputs you enter; real experiments can be distorted by selection bias, novelty effects, peeking, and non-independence between visitors. Always validate significant results with proper sample-size planning and guardrail metrics before rolling out changes.