You have changed something on your website, maybe a headline, a button colour or a checkout step, and you are watching two versions of the page fight it out. Version A gets a few more sign-ups than version B this week. The question that actually matters is whether that gap is real, something you could rely on if you rolled it out to everyone, or whether it is just the ordinary wobble you would see even if both versions were identical. That question is what an A/B test significance calculator answers.
Why a difference in the numbers isn't the same as a real result
Small samples are noisy. If you show two near-identical pages to a hundred visitors each, you will almost never get exactly the same number of conversions on both, purely by chance. Someone has to come out ahead on the day, even if there is nothing in it. The danger is treating that lead as proof and rebuilding your site around whichever version happened to win this particular week.
This is the problem significance testing was built to solve. It takes the numbers you actually saw, the size of each group and how many converted, and works out how likely it is that a gap that size would show up just from chance alone. If that likelihood is low, you have a result worth acting on. If it is not, you have noise dressed up as a winner.
What "statistically significant" actually means
A conversion rate is simply the share of visitors who did the thing you wanted, bought something, signed up, clicked through, out of everyone who saw the page. Comparing two conversion rates on their own tells you which number is bigger. It does not tell you whether that difference is reliable or a fluke.
Significance testing adds a second figure alongside the conversion rates: the probability that you would see a gap this large (or larger) if there were genuinely no difference between the two versions. Testers usually call the point where they stop trusting a result "confidence", most often set at 95%, meaning you are accepting a one-in-twenty chance of being wrong when you call a winner. The lower that probability figure, the more confident you can be that version A really does outperform version B.
How the calculator works
You give it four numbers: the number of visitors shown version A, the number who converted on version A, the number shown version B, and the number who converted on version B. It calculates the conversion rate for each version, then runs the statistical comparison and reports back whether the difference clears the significance threshold.
What you put in
- Visitors to variant A and the conversions it produced
- Visitors to variant B and the conversions it produced
What comes back
- The conversion rate for each variant, so you can see the raw gap
- Whether the difference is statistically significant at the standard confidence level
- How wide the sample still needs to be before a smaller gap would count as reliable
None of this involves guesswork on your part. The calculator is doing the arithmetic that would otherwise mean sitting down with a statistics textbook, and it shows the conversion rates alongside the verdict so you can see where the numbers actually came from.
A worked example
Say you tested a new call-to-action button against the original. The original (version A) was shown to 1,000 visitors and converted 40 of them, a 4% conversion rate. The new version (version B) was shown to 1,000 visitors and converted 52, a 5.2% conversion rate. On the raw numbers, version B looks like a solid win, nearly a third more conversions.
Run those four figures through the calculator and it checks whether a gap of that size, from samples of that size, is likely to be a genuine effect or something that could plausibly have happened by chance. With 1,000 visitors per side, a gap of 4% versus 5.2% often does clear the standard significance threshold, but the exact answer depends on the actual split of conversions, which is exactly why you run the numbers.
If the same gap had come from 100 visitors per side instead of 1,000, the calculator would most likely tell you it is not significant yet, even though the percentages look identical. Sample size changes how much you can trust a given gap, and that is the part a quick glance at two percentages will never show you.
Where this goes wrong
Stopping the test the moment one version pulls ahead
Checking your results every day and declaring a winner the first time version B looks better is one of the most common ways to fool yourself. Early in a test, small samples swing around a lot, and whichever version happens to be ahead on day two can easily fall behind by day ten. Decide on a sample size or a time period in advance and let the test run its course before you check significance.
Testing with too few visitors
If your site gets modest traffic, you may simply not have enough visitors to reach a reliable verdict in a reasonable time. A gap that would be obvious with 10,000 visitors per variant can sit in "not significant" territory indefinitely with 200. In that case, it is often more useful to test bigger, more obvious changes, where the effect is large enough to show up even in a smaller sample.
Running several variants and comparing them all against each other
The more comparisons you run at once, headline A versus B versus C versus D, the more likely you are to find one "significant" difference purely by chance, simply because you gave chance more chances. If you are testing more than two versions, treat any single winner with a bit more caution than you would in a straight two-way test.
What the calculator can't tell you
A significant result tells you the difference is unlikely to be chance. It does not tell you why version B performed better, whether that reason will hold up next month, or whether the same change would work on a different page of your site. It also cannot correct for a test that was set up badly, run over an unrepresentative period, such as the week of a one-off promotion, or ended early. The number it gives you is only as good as the data and the test design behind it, and it is worth treating the output as a strong signal to weigh up.
Putting a result to use
A/B testing is one tool among several for working out where your promotion effort is best spent, and it tends to matter most once you already have a reasonable flow of visitors to test with. If you are earlier than that and still deciding which channels deserve your time in the first place, the Promotion Channel Prioritiser works through that question directly. For the fuller picture of what to do before and after you have traffic to test, the Website Promotion Playbook sets the stages out in order, and the promotion glossary is there if a term anywhere in this guide needs unpacking further.