> For the complete documentation index, see [llms.txt](https://docs.bloop.plus/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bloop.plus/referral-program/ab-testing.md).

# A/B Testing

Guesswork is the most expensive way to set a reward. A discount that feels too small never spreads; one that feels too generous quietly eats your margin — and from the outside the two look the same. A/B testing replaces the guess with evidence: run two reward setups side by side on real referrers and let conversions decide which one to keep.

In Bloop, a test splits new participants into two groups. **Variant A** is a snapshot of your current rewards — the control you leave untouched. **Variant B** is a copy you edit to try one new idea: a larger discount, a different reward type, or a changed minimum requirement. Bloop assigns each new participant to A or B, tracks both, and when you end the test you declare a winner. If B wins, Bloop promotes its reward settings onto your live campaign automatically. You run tests from the **A/B testing** area of the **Campaign** tab, and only one test can be active per campaign at a time.

> **Best practice:** Test the change with the most money riding on it first. The reward value and the referee discount move conversions more than wording or layout, so settle those before fine-tuning anything smaller. Change one thing per test, let it run until both groups have real volume, and bank each winner before starting the next — that is how a program compounds instead of churns. For the wider playbook, see [referral program best practices](https://bloop.plus/blog/referral-program-best-practices/).

### What you can test

A Bloop A/B test compares two reward setups. Before you start, write your hypothesis as a single sentence you can prove true or false with the numbers. Change exactly one variable per test.

| What you can test              | The variable on Variant B                                   | A hypothesis to settle                                                                                                                           |
| ------------------------------ | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Reward value**               | Raise or lower the discount/credit/cash amount              | "Does a $15 referee discount convert enough more than $10 to justify the extra $5 we give away on every order?"                                  |
| **Reward type**                | Switch between a percentage discount, store credit, or cash | "Does store credit drive more referrals than an equivalent percentage discount, because it pulls people back for a second purchase?"             |
| **Referee vs referrer reward** | Move the generosity to one side of the give-and-get         | "If we put more of the budget on the referee side, do more first-time buyers convert — even if it makes referrers slightly less eager to share?" |
| **Minimum requirement**        | Add, remove, or change a minimum spend to qualify           | "Does a $50 minimum-order requirement protect our margin without scaring off referees, compared with no minimum at all?"                         |

The reward value and reward type are the heavy levers — they change both your conversion rate and your cost per acquired customer, so they deserve the first tests. A minimum requirement is more of a guardrail: it rarely lifts conversions, but it can stop the program losing money on tiny orders. Decide what you are optimising for before you start, because "more referrals" and "more profitable referrals" can point at different winners.

If you have not set up the rewards you want to compare, do that first in [Referrer and referee rewards](/referral-program/rewards.md) — Variant B is built as a copy of those settings, so the cleaner your starting point, the cleaner the test.

### What tends to win

You do not have to start from zero. A handful of patterns hold up across referral programs, and they are useful as the *hypothesis you test first* — not as a result you can skip testing. Your store, your margin, and your customers are the tiebreaker, so treat the points below as informed starting bets.

**Test the thing with the most money attached first.** Reward value moves conversions and cost more than anything else, reward type comes next, and wording or layout last.

On offer structure, a two-sided **give-and-get** reward — where both the referrer and the referee get something — consistently outperforms a referrer-only offer. It lifts conversions on both sides of the transaction and tends to improve customer lifetime value. Counter-intuitively, more generous is not automatically better: testing a lower incentive sometimes wins on total economics because you acquire customers more cheaply.

If you extend testing to the creative and layout of your share experience, simpler and product-led tends to win:

| Element          | What to try                     | What tends to win                                         |
| ---------------- | ------------------------------- | --------------------------------------------------------- |
| Reward structure | Referrer-only vs give-and-get   | **Give-and-get** — lifts LTV and conversion on both sides |
| Background       | Branded/colourful vs plain      | **White background**                                      |
| Imagery focus    | Lifestyle/people vs the product | **Product-focused creative**                              |
| Human imagery    | People in the creative vs none  | **No-human design**                                       |
| Trust cues       | No social proof vs social proof | **Social proof present**                                  |

> The patterns in the table above are directional starting bets, not measured Bloop results. Use them to choose what to test first; let your own Bloop data decide what you actually keep. What wins for a fashion brand can lose for a supplement brand, which is exactly why you run the test instead of trusting the chart.

### How long to run a test, and how to read the results

The honest answer to "how long?" is **until both variants have enough conversions to trust the gap between them** — which is about volume, not the calendar. A test that has run for three weeks but only collected a handful of conversions per side has told you nothing; a high-traffic store can reach a trustworthy read in days.

A few rules of thumb keep you out of trouble:

* **Do not conclude from a handful of conversions.** With five conversions on one side and three on the other, the "winner" is almost certainly noise. You want both variants to have built up real, comparable totals before you compare their rates.
* **Do not stop the moment one side looks ahead.** Early in a test the lead bounces back and forth — this is normal. Stopping the instant Variant B pulls ahead ("peeking") is the single most common way to crown a false winner, because you are ending on a lucky swing rather than a stable result.
* **Look for a gap that holds.** A difference you can trust is both large enough to matter to your business and stable over time. If A and B are within a whisker of each other after good volume, that is a real result too: it means the change you tested did not matter, so keep the cheaper option.
* **Watch the cost, not just the count.** A variant can win on raw referrals and still lose money if it gives away more per order. Read the enrolled counts and conversion metrics alongside what each reward costs you — the goal is more *profitable* referrals, not just more of them.
* **Lower-traffic stores need patience.** If your referral volume is thin, a test simply needs to run longer. If volume is so low that you would wait months, A/B testing is the wrong tool for now — focus on driving more referral traffic first.

### A worked example

Here is the full loop from question to decision. *All numbers below are illustrative — they are an example of how to reason, not measured Bloop results.*

1. **Start with a hypothesis.** Your live offer gives referees a 10% discount. You suspect 15% would spread further, but you are not sure the extra giveaway pays for itself. Hypothesis: *"A 15% referee discount will lift the referral conversion rate enough to beat the higher cost per order."*
2. **Set up Variant B.** Bloop copies your current rewards into B. You change one thing: the referee discount from 10% to 15%. Variant A — the 10% control — stays untouched. You name the test "Referee 10% vs 15%" and start it.
3. **Let it run.** Bloop assigns each new participant \~50/50 to A or B and keeps them there. You leave it alone and resist checking hourly. After a few weeks each side has gathered a comparable, sizeable batch of conversions.
4. **Read the results.** Suppose Variant B (15%) converts referees at, say, 12% versus Variant A (10%) at 9% — a clear, stable gap after good volume. B wins on conversions. But B also gives away 50% more discount per order, so you check the economics: are those extra conversions worth the extra margin you spent? In this example the higher conversion rate more than covers the deeper discount, so B is the real winner on profit too.
5. **Decide and bank it.** You declare **Variant B** the winner. Bloop copies B's 15% referee discount onto your live campaign automatically, and it becomes your new default. You have turned a hunch into a banked improvement — and your next test can start from this stronger baseline.

If the numbers had come back the other way — B converting only marginally better while costing far more per order — the right call would be to declare **Variant A** the winner, keep your 10% offer, and test a different idea next.

### How variants are created

When you create a test, Bloop builds it from your existing campaign:

* **Variant A** points at your current referrer reward and referee reward. It is the control — you do not edit it inside the test.
* **Variant B** is a fresh copy of those same rewards. This is the variant you change to test a new idea.

Because B starts as an exact copy of A, the only difference between the two groups is whatever you change on B. That keeps the comparison clean.

### How participants are assigned

Bloop assigns each new participant to a variant and keeps them there for the life of the test:

* Assignment is **roughly 50/50**. Bloop hashes a stable identifier (the customer ID, or a browser cookie for logged-out visitors) together with the test ID, so the same person always lands in the same group.
* A participant who already has an assignment keeps it. Logging in, switching devices, or returning later does not move them to the other variant — their customer ID and cookie are reconciled to the existing assignment.
* Customers who became referrers **before** the test started are excluded. They keep your normal campaign rewards and are not counted in the test.

This means a referrer always sees and shares the reward for their assigned variant, and their referees receive the matching referee reward.

### Run an A/B test

1. Open the **Campaign** tab and go to **A/B testing**.
2. Create a new test. Bloop generates a draft with Variant A (your current rewards) and Variant B (a copy).
3. Edit **Variant B** — change the referrer reward, the referee reward, or both. Adjust the discount value, switch the reward type, or change a minimum requirement to form your hypothesis. Variant A stays untouched.
4. Give the test a name and start it. The test moves from **draft** to **active** and Bloop begins assigning new participants.
5. Let the test run long enough to gather meaningful data for both variants.
6. When you are ready, open the test and review the enrolled counts and metrics for A and B.
7. Declare the winner — **A** or **B** — to end the test.

### What happens when you declare a winner

Ending a test moves it to **inactive** and records the result:

| Winner    | What Bloop does                                                                                                      |
| --------- | -------------------------------------------------------------------------------------------------------------------- |
| Variant A | Your campaign keeps its current rewards. Variant B is discarded.                                                     |
| Variant B | Bloop copies Variant B's reward settings onto your live campaign rewards, so the winning setup becomes your default. |

In both cases Bloop stores a snapshot of each variant's rewards, enrolled count, and metrics, then clears the test's assignments and the temporary Variant B reward records. You can reopen the finished test later to review the snapshot.

### Common mistakes to avoid

* **Changing more than one thing at a time.** If Variant B has a bigger discount *and* a different reward type, a win tells you nothing about which change caused it. One variable per test, always.
* **Stopping too early.** A few conversions on each side is noise, and ending on an early swing crowns a false winner. Wait for real, comparable volume before you decide.
* **Testing the small stuff first.** Tweaking wording before you have settled the reward value is optimising the cheap lever while ignoring the expensive one. Test the change with the most money attached first.
* **Ignoring margin.** A variant that wins on raw referrals can still lose money if it gives away more per order. Judge winners on profitable referrals, not headline counts.
* **Testing when volume is too low.** With thin referral traffic, no test ever reaches a trustworthy read. If you would have to wait months, grow your referral volume first and test once you have the traffic to support it.

### Next steps

* Set up the rewards you will be testing in [Referrer and referee rewards](/referral-program/rewards.md).
* Review your campaign rules in [Create and configure a referral campaign](/referral-program/campaign.md).
* Track the outcome of a winning test in [Referral analytics](/referral-program/analytics/analytics.md).
* Put your winners to work with the wider [referral program best practices](https://bloop.plus/blog/referral-program-best-practices/) playbook.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.bloop.plus/referral-program/ab-testing.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.