You can run A/B tests to test different elements of an email campaign to make data-driven decisions about which areas of an email are performing best. Send different variants to a subset of your audience to compare results to better understand performance.
Suggested uses for A/B tests:
Testing subject line
Product recommendations
Promotional offers
Static content
Show or hide prices on items
Different hero images
Different call-to-action copy
Two different test types can be run within Bluecore campaigns:
A/B test
An A/B test can be configured on both one-time and automated campaigns and allows you to manually choose a winner.
Auto-Winner A/B test
Auto-Winner tests can only be configured on one-time campaigns and allows for an automatic winner selection.
NOTE: A/B or Auto-Winner A/B tests are unavailable when send time optimization (STO) is used.
Setting up an A/B Test
To set up an A/B Test, build your campaign normally, following instructions in the Campaign Builder guide.
When you get to the message tab, you have the option of making the campaign an A/B Test.
Below is an overview of the options when setting up an A/B Test:
A/B Test Name
Name your A/B Test. You can name it based on the type of test you’re running, such as “Offer Test” or “Cart Abandon Subject Line Test”
Test Type
Optional field
Elements of the campaign you plan to test. This has no impact on the campaign settings or test, but could help with analyzing your test results.
Incentives
Change the motivation of the campaign, such as free shipping against a discount
Recommendations
Change the recommendations shown to the customer
Creative
Change the look of the test campaigns, such as a hero image
Subject Line/Copy
Change the text of a campaign, either in the subject line or body of the email
Personalization
Test using different customer attributes in a campaign
Custom
A custom A/B Test
Winning Criteria
Optional field
The metric that determines a successful test
Conversion Rate
The number of customers who purchased, divided by the number of customers who received an email.
Click Through Rate
The number of customers who clicked, divided by number of customers who received an email.
Test Description
Optional field
Use this field to add any additional information about your test parameters, such as the reason for the current test, expected results, or any other relevant considerations.
Completing the Auto-Winner A/B Test setup
If you’re creating an Auto-Winner A/B Test through a one-time send campaign, complete these additional settings:
Holdout Group
The Holdout Group is the percentage of people that will not receive the initial campaign and will be sent the winner when the test is complete. Bluecore recommends setting this percentage to 70-80%.
This number must be between 10-90%.
Defaults to 90%
Test Duration
Determine a test duration to determine how long the test will run for. This number must be between 3 hours and 10 days.
Defaults to 3 hours.
Declare a Winner
If this setting is turned on, Bluecore determines a winning variant once 95% confidence is reached and sends the winner to the holdout group. If Bluecore does not reach 95% confidence, the test runs for the full duration and then the winner gets declared and sent when the test is complete.
Defaults to Off.
Creating the variants
The next step is to create the variants that are being tested. Up to five variants may be added.
Variant Percentages
Enter a percentage of the audience to which this variant will be sent to. The percentages must add up to 100%. Click Re-balance to evenly distribute the percentages.
Variant Name
This is an internal only name that’s used for organization and analytics.
Variant Subject Line
Preview Text
Add any variant-specific tracking parameters. Tracking parameters that are configured in the marketing settings are automatically added here.
Configure the email template. Create a new template, select from a pre-built template, or copy an existing template. For more on email templates, navigate to VTE.
NOTE: Be sure to configure the applicable recommendations rules outside of the template once the email is created if product recommendations are being tested.
After the first variant is set up, set up Variant B.
NOTE: Click Copy from Variant and select a variant to make configuring the settings and content faster.
Click Save & Continue after all variants have been set up.
Analyzing your A/B Tests
The best way to analyze your tests is through the Experimentation Hub, which allows you to review and manage your campaigns’ A/B Tests in one centralized location within Bluecore.
Experimentation Hub optimizes your time spent on testing by letting you know when a test is ready to declare a winner.
For more information on analytics for A/B Tests, see Experimentation Hub.
Selecting a winner
When an A/B Test has run long enough to achieve statistical significance, the next step is to select a winner for the campaign. By selecting a winner, the remaining recipients will receive the email campaign that performed better.
By selecting a winner, this campaign will turn into a single-content version, sending the version that has been selected as the winner. The A/B versions will no longer be available.
Select a winner by completing the following steps:
Hover over the flag
in the sidebar menu and select Automated Campaigns or One-Time Campaigns, depending on the type of campaign you’re working on. The list of Communicate Campaigns displays.Navigate to the active campaign with the A/B Test within the campaign.
Click the A/B Test or Auto-Winner A/B Test tab at the top of the campaign.
Click the Select as winner button on the applicable variant.