Home / A/B tests
Find the widget that converts best.
Put 2 to 4 of your widgets head to head on the same site. Visitors are split at random, the dashboard reports what each one earns, and tells you which one to keep. It also tells you when it is too early to say.
- Each visitor always sees the same widget
- No verdict on thin data
- Results by email
Variant B is likely the best (96% chance).
| Variant | Visitors | Conversions | Chance to be best |
|---|---|---|---|
| AAnnouncement bar | 4,210 | 188 (4.5%) | 4% |
| BEmail popup | 4,198 | 251 (6.0%) | 96% |
| ControlNo widget | 930 | 29 (3.1%) | - |
Example figures for illustration. Your dashboard shows your own numbers.
How a test works
- Pick 2 to 4 widgets. They must be active and belong to the same campaign. Their targeting (countries, languages, devices, browsers, schedule) has to match so every variant reaches the same visitors. Page rules and triggers can differ, with a warning.
- Choose what counts as a win. A widget action or a conversion URL, explained below.
- Choose whether to measure incrementality. Optional, off by default.
- Set the length and the ending. Pick 7, 14 or 30 days, or an end date, and what happens when the test ends.
Before you start, a summary restates everything in one paragraph. While the test runs, each tested widget carries an A/B Test badge in your widget list, linking to the results.
A widget can be in one running test at a time. Changing a tested widget mixes old and new results, so the editor warns you. Pausing or deleting a tested widget ends the test.
Two ways to measure a win
- Widget action
- A click on the widget's button, an email or a form submitted, or feedback given. Closing the widget never counts as a win. We record closes separately and show the share of visitors who closed each widget.
- Conversion URL
- A visit to a page such as your thank-you or order confirmation page. Enter one or more addresses, one per line. A page counts when its address contains your text, capitals ignored, so
merci/matches/fr/merci/. The GrowthTools pixel must be installed on that page. Add an average value per conversion to see results in money.
Comparing different widget types? A top bar against an email popup is allowed. Use a conversion URL, so both are judged on the same result rather than on different kinds of click.
Find out what your widgets really add
A test tells you that B beats A. It cannot tell you whether either one brings anything at all. Maybe those visitors would have bought anyway.
Turn on the control group and 10% of visitors see none of the tested widgets. Say 5 in 100 visitors who see a widget buy, against 3 in 100 who see nothing. Your widgets add 2 sales per 100 visitors. Those are your incremental conversions.
With an average value per conversion, the top of your dashboard shows one line: This month: +$7,965 incremental revenue (estimated). It is an estimate from your own numbers, and says so. When the data is too thin, it says too early to tell instead of showing a figure.
When to leave it off. The control group needs a goal that visitors can reach without the widget, such as a purchase. If the widget is the only way in, for example a gated download, leave it off. A visitor who sees nothing cannot click anything.
Numbers you can trust
Each visitor gets a random identifier and is assigned a variant once. They see that same widget on every page and every visit, so results are not mixed.
The chance to be best comes from Bayesian statistics. "96%" means that, given your data, B is the best widget in 96 out of 100 plausible scenarios. It is easier to read than a p-value.
Early numbers move a lot, so we hold back. A winner is named only when all three conditions are met:
| Condition | Minimum |
|---|---|
| Test duration | 7 days |
| Results per widget | 30 |
| Chance of being the best | 90% |
Until then the page reads Not enough data yet, or No clear winner yet when two widgets are close. When you create the test, we estimate how long you need from your recent traffic and warn you if your chosen duration looks too short.
When the test ends
You choose the ending when you create the test, and can change it while the test runs.
- Pause the losing widgets
- The winner keeps running alone and the test closes. Without a clear winner, nothing is paused, the test keeps running and the email tells you so.
- Keep the test running
- Same split, same control group, and the results keep updating. You can stop it any time.
- Pause all tested widgets
- Always applied at the end date, winner or not.
Then we email you: the verdict in one sentence, the results table, the incremental figure if you used a control group, and the action we applied (for example "As requested, we paused Variant A. Variant B keeps running."). If visitors without a widget converted more than those with one, the email says so plainly.
On the results page, Keep this one picks a winner by hand and pauses the others. Stop test ends the test and pauses all tested widgets.
Privacy and cookie consent
The test stores a random identifier in the visitor's browser, on your own site. It holds no personal data, it is never shared between sites, it expires after 13 months, and it is used for statistics only. You stay responsible for your cookie banner.
If you prefer, tick Wait for consent on a test. Your banner then calls growthtools('consent', true) when the visitor agrees. Until then, visitors see Variant A and are not counted.
Plans
| Plan | A/B tests |
|---|---|
| Free | Not included |
| Starter | 1 running test at a time |
| Growth | Unlimited |
Finished tests do not count toward the limit.
Questions
Long enough to get a representative sample. The test form estimates the time from your own traffic. We name a winner only after 7 days, with at least 30 results per widget and a 90% chance of being the best. Until then the results page says there is not enough data.
Yes. You get a warning because they do different jobs. Judge them on a conversion URL, such as your thank-you page, so both are measured on the same result.
An option for tests that use a conversion URL. One visitor in ten sees none of the tested widgets, so you can see what the widgets add compared with showing nothing. Leave it off when the widget is the only way to reach the goal.
We say so. With no clear winner, nothing is paused (unless you chose to pause all) and the test keeps running. The email explains it and you can pick a winner yourself.
The test uses a random identifier stored in the visitor's browser on your site, with no personal data. You stay responsible for your own cookie banner. You can also tick Wait for consent, and visitors are then counted only after your banner gives the go-ahead.
Keep the widget that earns the most.
Build two widgets, start a test and let your visitors decide.
Create your free account