How to A/B test your referral rewards

A reward too small never spreads, and one too generous eats margin. An A/B test runs two reward setups side by side so conversions, not guesses, pick your reward.

What you’ll set up

Control (A): your campaign’s current rewards.
Variation (B): a copy you change to test one idea, such as a bigger discount or another reward type.
Test length: 2 weeks, 4 weeks (recommended) or 8 weeks. Bloop gives the final read when it ends.
Main metric: the number the test is judged on, Customers with a referral order or Customers who shared.
New referrers are split 50/50 and stay on their side for the whole test. Referrers who joined before the test keep the campaign rewards.

How to run an A/B test

Create the test → Click Start test
1. Open Referral program and open your campaign.
2. Click Overview. The A/B Testing section lists the campaign’s tests:
3. Click Create test. If you started one before, the button reads Continue setup:
4. Enter a Test name.
5. Under Variation, click Edit reward on the referrer or referee reward and change one thing. Edit reward under Control changes your live campaign rewards. Bloop won’t start a test where B has the same rewards as A.
6. Pick the Test length.
7. Pick the Main metric. Shares reach a result faster when your program has few referral orders. Below, Bloop estimates the smallest difference your traffic can detect: see What the estimate means.
8. Click Start test:
A/B tests can’t run with FlexiTiers. If the campaign has two or more tiers, remove them first.

While the test runs, the campaign’s Referrer reward and Referee reward are read-only. End the test to edit them.
Read the results → End the test
1. On the A/B Testing page, click your test.
2. Read the result on your main metric first. The page compares A and B on it and says how likely each side is to be better. Then check Reward cost and Revenue after rewards under Cost (not used to pick a winner):
3. When the test length ends, Bloop emails you that the result is ready. Click Review and end test. If you end earlier, Bloop warns you first: early leads often shrink.
4. Choose Control A or Variation B.
5. Click End. Bloop applies the winner’s rewards to your campaign. Rewards already given stay as they are; from then on everyone in the campaign, including customers who were in the other group, earns the winner’s rewards:
An ended test can’t be restarted, so don’t end one just to look. Results stay on the page.

If you don’t pick a winner

When the test length ends, the test page shows a warning with the date the test will end on its own, and a shortcut to Review and end test.

If you don’t pick a winner within 14 days of the email, Bloop ends the test for you and keeps Control A, so your reward settings don’t change. The test is labelled Ended automatically, the page notes that Control A was kept, and you get a second email.

Running tests also end, keeping Control A, if the app is uninstalled. The test page notes why it ended.

What the estimate means

The estimate uses the customers who saw your referral program in the last 28 days. For example: “With 4 weeks, the test can tell A and B apart if one does at least 31% better than the other, for example 3.8% vs 5.0%.” The difference is relative: if 3.8% of customers in Control A share, Variation B has to reach about 5.0% for the test to see it.

A smaller number means a more sensitive test, so a longer test or a more frequent metric helps. The estimate updates as you change either one.

Your store may be too small to measure this metric: even an 8-week test would only detect a difference above 50%. For referral orders, Use “Customers who shared” switches the main metric in one click. Otherwise, grow referral traffic first.
Not enough recent traffic: fewer than 50 customers saw your referral program in the last 28 days, so there is nothing to estimate from yet.

What the results mean

Bloop gives a read-out after 100 customers per variant and 7 days. Until the test length ends, the page shows the leader so far and the day you are on.

Metric	What it counts
Customers with a referral order	Share of the group’s customers whose friends placed a first order
Customers who shared	Share of the group’s customers who clicked a share channel or copied their link
Referral revenue	Revenue from referred friends’ first and repeat orders during the test, net or gross as labelled
Referral invites	Times referrers clicked a share channel or copied their link
Referee store visits	Clicks on referral links
Referee coupon claims	Coupons claimed by referred friends
Referral orders	First orders placed by referred friends during the test
Orders per invite	Referral orders ÷ referral invites
Reward cost	Discounts used by referred friends plus rewards given to referrers for first orders during the test
Revenue after rewards	Referral revenue minus the cost of referrer rewards. Discounts used by referred friends are already taken out of referral revenue, so they are not subtracted again

Judge the test on the main metric: revenue can swing on one big order. Then check the cost, because a variant can win on referrals and still lose money by giving away more per order. Reward cost and Revenue after rewards sit under their own Cost (not used to pick a winner) heading and are never colored as better or worse: a higher cost can still be worth it. Custom rewards have no money value, so they are not included in the cost.

Best practices

Change one thing per test. If B changes two things, you won’t know which one worked.
Test value and type first. They move conversions more than anything else you can change.
Let it run for the test length. Early leads swap back and forth. “No difference” is a result too: keep the cheaper reward.
Low traffic? Check the estimate before you start. If no test length can detect a realistic difference, grow referral traffic first.

Referrer and referee rewards - the settings you test
Reward display - how the reward reads to customers
Referral analytics - track the result after the test