Understanding A/B test results
A test result is only worth acting on if you can trust it. Bloop reads every A/B test with Bayesian statistics, a method widely used by A/B testing tools because it answers the questions a business actually asks: how likely is B to win, and what does a wrong pick cost? This page explains how that works, what each number on the results page means, and what Bloop does to stop you from switching rewards on luck. You don’t need any statistics yourself: Bloop writes every result as a plain sentence. To set up a test, see How to A/B test your referral rewards.
The question every test answers
Section titled “The question every test answers”Two groups never end with exactly the same numbers, even when both get the same reward. Some customers share more than others, and chance decides who lands in which group. So “B has more orders” isn’t enough to change your reward. What you need to know is:
- How likely is it that B is really better than A?
- By how much?
- What do I lose if I pick B and I’m wrong?
Bloop’s Bayesian method answers all three directly, with a chance such as “98.3% chance B beats A” instead of a technical score you have to interpret.
How Bloop calculates the answer
Section titled “How Bloop calculates the answer”Bayesian A/B testing treats each reward’s real performance as something to learn from your customers, step by step:
- It starts neutral. Before the first customer arrives, A and B are treated as equal. Bloop doesn’t favor your current reward or the new one, so the result comes only from your customers.
- It learns from every customer. Each customer in the test sharpens Bloop’s estimate of how well each reward really performs. With few customers the estimate is broad; with more it narrows.
- It compares A and B directly. From the two estimates, Bloop calculates the chance that B really performs better than A, and the likely size of the gap. The chance is calculated exactly rather than by random simulation, so the same data always gives the same answer.
- It weighs the downside. Bloop calculates the expected loss of picking the leader: how much you would give up on average if it turns out to be the worse reward. You get the odds and the risk side by side.
How customers are counted
Section titled “How customers are counted”- Each customer who sees your referral program during the test is placed in A or B at random, like a coin flip, and stays on that side for the whole test. Both groups shop in the same weeks, so sales, seasons and ads affect them equally.
- Customers who joined your program before the test keep the campaign rewards and are not part of the test.
- Bloop compares customers, not clicks or orders. The main metric is the share of each group’s customers who did something: had a friend place a referral order, or shared. A referrer who brings in 20 orders counts once, so one very active customer can’t decide the test alone.
What the numbers mean
Section titled “What the numbers mean”This is how a finished test can read on the results page, and what each line tells you:
| On the results page | What it means |
|---|---|
| 98.3% chance B beats A. | Based on the customers so far, B is the better reward 98.3 times out of 100. In the other 1.7, A is. The chance is never 100%: a test makes you more sure, never certain. |
| B is clearly better. | The verdict. Bloop says clearly at a 95% chance or more and likely from 90% to 95%. Below 90% it says No clear winner yet. |
| Customers with a referral order: A 120 of 2,400 (5.00%), B 154 of 2,400 (6.42%) | The main metric for each group: how many customers did it, out of everyone in that group. |
| Likely difference, B minus A: +0.10 to +2.73 percentage points (95% range) | How big the gap probably is. The true difference is very likely inside this range: here, B is ahead by somewhere between 0.1 and 2.7 points. A narrow range means you know the size of the gain. A wide range means B is probably better, but by an uncertain amount. A range that sits evenly around 0 means there may be no real difference. |
| Risk if you pick B now: very small (under 0.01 percentage points) | What you would lose on average by switching to B if B is actually the worse reward. It weighs how likely a wrong pick is against how costly it would be. A risk far below the likely gain makes B a safe bet. |
Why Bloop waits for the test length
Section titled “Why Bloop waits for the test length”Early numbers swing. In the first days, a handful of customers can push either side ahead, and the lead often shrinks or flips as more customers arrive. If you check every day and stop the moment one side looks good, you will often lock in a lucky streak instead of a better reward.
So you pick a test length when you start: 2, 4 or 8 weeks. Whole weeks give every day of the week equal weight, so weekend shoppers count as much as weekday ones. Until the length is over:
- When one side reaches a 90% chance, the page says B is ahead so far (or A) with the day count, for example Day 9 of 28, never “clearly better”.
- If you end the test anyway, Bloop warns you first: “The planned test length isn’t over yet. Early leads often shrink, so this result may change.”
Look as often as you like: looking doesn’t change the result. Acting on an early lead is what’s risky.
The safeguards built into every test
Section titled “The safeguards built into every test”| Safeguard | What it protects you from |
|---|---|
| A minimum before any result. No read-out until each group has 100 customers and the test has run for 7 days. | A verdict built on a handful of customers |
| A fixed test length. Before it ends, the leader is only “ahead so far”. | Stopping on a lucky streak |
| A fair-split check. A and B should get about the same number of customers. If the split is so uneven that chance would explain it less than 1 time in 1,000, Bloop hides the result and shows a warning. | Trusting a test whose 50/50 split went wrong |
| Customers, not orders. Each customer counts once on the main metric. | One big referrer or one large order deciding the test |
| Cost kept apart. Reward cost and Revenue after rewards sit under Cost (not used to pick a winner). | Revenue, which a single order can swing, picking your winner |
| An estimate before you start. | Running a test your traffic can’t answer |
| No silent switch. If you don’t pick a winner within 14 days of the results email, Bloop ends the test and keeps Control A. It never switches to B for you. | A reward change you didn’t approve |
How to act on each result
Section titled “How to act on each result”| The page says | What it means | What to do |
|---|---|---|
| Not enough data yet | Fewer than 100 customers in a group, or under 7 days | Keep the test running |
| B is ahead so far (or A) | One side has a 90% chance or more, but the test length isn’t over | Keep running. Early leads often shrink |
| No clear winner yet, before the end | Neither side has reached 90% | Keep running |
| B is clearly better | A 95% chance or more when the test length is over | Pick Variation B after a look at Reward cost |
| B is likely better | 90% to 95% when the test length is over | B is a sound pick, but A is still the better reward up to 1 time in 10. Check the risk and the cost |
| A is clearly better or A is likely better | Your current reward wins | Pick Control A |
| No clear winner yet, after the end | Neither side reached 90% | See When there’s no clear winner |
| A and B received very different numbers of customers | The split may be broken, so the result isn’t reliable | Don’t pick a winner on this result. Contact support |
When there’s no clear winner
Section titled “When there’s no clear winner”“No clear winner” at the end is a real answer, not a failed test. Either the two rewards perform about the same for your customers, or the difference is smaller than your traffic can detect. The 95% range tells you which:
- Narrow range around 0: the rewards really are about equal. Keep the cheaper one: compare Reward cost.
- Wide range: your store didn’t have enough customers to tell. Next time, test a bigger change, pick a longer test length, or use Customers who shared, which happens more often and reaches a result faster.
What the estimate tells you before you start
Section titled “What the estimate tells you before you start”The estimate on the Create page tells you the smallest difference your traffic can reliably detect in the test length you picked, for example “at least 31% better”. If the real difference between your two rewards is smaller than that, the test will most likely end with no clear winner. See What the estimate means.
Limits to keep in mind
Section titled “Limits to keep in mind”- A test tells you which reward worked better for your customers during the test. A result from a big sale period may not hold in a quiet month, and the other way round.
- Likely is not certain. At a 90% chance, A is still the better reward 1 time in 10. That’s why Bloop shows the risk next to the chance.
- The verdict covers the main metric only. Check Reward cost and Revenue after rewards before you switch.
Why is the chance never 100%? A test sees a sample of your customers, not every customer you will ever have. More customers shrink the doubt but never remove it.
Can I check the results every day? Yes. Looking doesn’t change anything. Just don’t end the test on an early lead.
B has more referral orders. Why is there no winner? A gap of a few orders is normal between two random groups, even when both get the same reward. Bloop names a winner only when the gap is too large to be chance.
Why Bayesian statistics? It answers the question you actually ask, “how likely is B to be better?”, in one number, and adds what a wrong pick would cost. Other methods give a score that only says whether a gap is unusual, which is easy to misread as the chance of winning.
Related
Section titled “Related”- How to A/B test your referral rewards - set up, read and end a test
- Referrer and referee rewards - the settings you test
- Referral analytics - follow your program after the test

