Running an A/B test allows you to display two or more different variants of a campaign to different sets of visitors at the same time. This test compares which variant drives has better performance (orders, emails captured).
GETTING STARTED
Before getting started with setting up an A/B test, please be sure to already have all campaign creatives created. When setting up an A/B test, you are able to select the previously created campaign creative. You are not able to create new campaign creative when configuring the A/B test in the following steps.
SETTING UP THE CAMPAIGN
After you’ve built the campaigns you’d like to test, the next step is to set up the A/B test.
Navigate to the flag icon on the left-hand navigation.
Go to Site.
Then Campaigns.
Click Create New Campaign in the upper right-hand corner.
Select A/B Test as the campaign type.
Click Next.
CAMPAIGN SETTINGS
1. Enter a Name for the A/B test campaign.
2. Determine when to start and end the campaign (Start at Launch, Start Date/Time, No End Date, End Date/Time).
3. Select a Display Time.
All Day: The popup will display continuously.
Specific Times: Determine a specific timeframe for the popup to display.
4. Configure your suppression rules. Suppression rules are the defined criteria that determine when a popup will and will not be shown to a shopper.
Show the popup until it has been displayed for: Determine the frequency of how many times the popup will be displayed until it is suppressed.
Show the popup until: Display the popup until it has been closed or engaged with.
Always show (no suppression rules): Always display the popup without any suppression rules.
5. At this time, the winner is determined manually. After the campaign has completed, please follow the instructions in the FAQ at the bottom of this article to select a final winner for the campaign.
6. Click Save & Continue in the lower right-hand corner.
TARGETING
Define your campaign targeting criteria that determine when to display a campaign.
1. Select which device to display a campaign on (Desktop or Mobile).
2. Then, create your own targeting criteria from scratch or from an existing rule.
CREATING A TARGETING RULE
Build out a target for your campaign. The targeting conditions include URL behaviors, visit frequency and engagement, date/time, and other conditions.
NOTE: Targeting rules will apply to all of the campaigns in the A/B test, as well as the holdout group. If a campaign has previously created targeting rules, they will be overridden with the targeting rules created here.
After you've built and saved your rules, click Save & Continue.
MESSAGE
The message section is where the variations are configured. Follow the below steps to configure each variation for the A/B test.
1. Enter the name for the variation.
2. Select the duration for the variant. Please be sure the durations for all variants are the same. For example, if you’re selecting “for campaign duration” for variant A, please select “for campaign duration” for all following variants to ensure the same duration.
For the campaign duration: Select this option to use the campaign duration configured on the A/B Test Settings screen.
For [X] days: Select a number of days to display the variation.
3. Click Select Campaigns. Select the previously created campaign creative here.
4. Click Select Campaigns to Add.
5. Enter a holdout percentage. A holdout group is a random selection of customers who are withheld from receiving any variant to compare performance of the holdout group to those who received any variant.
6. Repeat the above steps to add up to 4 variations (including a holdout). Additional variants may be added by clicking the Add Variant button.
7. Enter a percentage of recipients for each of the variants. The percentage must add up to 100%. Click rebalance to evenly distribute between the other variants.
8. Click Save & Continue in the lower right-hand corner.
SUMMARY
View an at-a-glance summary for all of the previously configured settings from the above steps.
Click Edit on any of the sections to adjust any of the previously configured settings.
SAVE AND LAUNCH
After you've reviewed all of your settings, targeting rules, and campaign message, click Launch Campaign in the lower right-hand corner to make your A/B test live!
NOTE: By launching the A/B test campaign, all campaigns with the associated creative are also launched. These campaigns will only be shown to users who meet the A/B testing targeting conditions and not individual customers. All associated parts of the A/B test campaign will be visible on the campaign dashboard.
ANALYTICS
After a campaign has launched, you can view the Site analytics for metrics and A/B test specific metrics.
FAQ
Q: Can I modify the holdout/variant percentage after the campaign has launched?
A: Pause the current A/B test campaign to modify the holdout or variant percentage. Repeat the above steps to edit the percentages. The percentages must add up to 100%.
Q: How do I launch the winning campaign after I’ve gotten my A/B test results?
A: After you’ve analyzed the results of different variants and determining a winner using the Site Analytics, pause the A/B test. Once the campaign has been paused, edit the A/B test campaign and set the winning variants percentage to 100%, and the other variants to 0%. Save the A/B test and launch the campaign again.
Q: Can I run an A/B test with only a single variant and a holdout?
A: Running an A/B test with a single variant allows you to test its relevance across a specifically held-out group of customers to better understand a lift within the specified campaign. Delete one of the variants and configure a holdout group.