Run an A/B tests to test different elements of an email campaign to make data-driven decisions about which areas of an email are performing best. Send different variants to a subset of your audience to compare results to better understand performance.
Suggested use cases:
Testing subject line, product recommendations, promotional offers, or static content
Testing open rate, click rate, conversion rate, click through rate, or revenue per email
Show or hide prices on items to see which performs better
Different hero images
Different call-to-action copy and colors on buttons
Two different test types can be run within Bluecore campaigns:
A/B test - An A/B test can be configured on both one-time and automated campaigns and allows you to manually choose a winner.
Auto-Winner A/B test - Auto-Winner tests can only be configured on one-time campaigns and allows a quick testing setup with an automatic winner selection.
GETTING STARTED
Get started by creating a campaign or editing a campaign. After you’ve configured the campaign settings, follow the below steps.
NOTE: A/B or Auto-Winner A/B tests are unavailable when send time optimization (STO) is used.
1. Select campaign type of A/B Test or Auto-Winner A/B test.
2. Select a Test Type of what’s going to be tested during this experiment: subject line, product recommendation, promotional offer, or static content.
3. Enter an A/B or Auto-Winner A/B Test Name. This is an internal only name that’s used for organization and analytics.
If you’re creating an Auto-Winner A/B Test, complete these additional settings:
1. Select the winning criteria from the drop-down. When the experiment is finished running, a winner is determined based on: open rate, click rate, conversion rate, click through rate, or revenue per email.
2. Enter a holdout group percentage. This number must be between 10-90%. The holdout group is the percentage of people that will not receive the initial campaign and will be sent the winner when the test is complete. Bluecore recommends setting this percentage to 70-80%.
3. Determine a test duration to determine how long the test will run for. This number must be between 3 hours and 10 days.
4. Turn declare a winner on or off. If this setting is turned on, Bluecore determines a winning variant once 95% confidence is reached and sends the winner to the holdout group. If Bluecore does not reach 95% confidence, the test runs for the full duration and then the winner gets declared and sent when the test is complete.
CREATING THE VARIANTS
After the above settings of the A/B or Auto-Winner A/B test have been configured, the next step is to create the variants that are being tested. Up to five variants may be added.
TIP: Click copy from variant in the upper right-hand corner and select a variant to make configuring the below settings and content faster and easier!
Enter a percentage of the audience to which this variant will be sent to. The percentages must add up to 100%. Click Re-balance to evenly distribute the percentages. For an Automated A/B test, these variants are split among the percentage that is not being held out. For example, if 80% is the holdout group and two variants are configured, split the remaining 20% between the two variants.
Enter a Variant Name. This is an internal only name that’s used for organization and analytics.
Enter a subject line and preview text for the variant. If this is what’s being tested, get creative!
Add any variant-specific tracking parameters. Tracking parameters that are configured in the marketing settings are automatically added here.
5. Next, configure the email template. Create a new template, select from a pre-built template, or copy an existing template.
6. After the email template is configured, click Finish and Close in the upper right-hand corner to save.
NOTE: Be sure to configure the applicable recommendations rules outside of the template once the email is created if product recommendations are being tested.
7. After the first variant is set up, set up variant B. Then, click + Add Variant to add up to five variants.
8. Click Save & Continue after all variants have been set up.
9. Then, Save & Launch the campaign!
SELECTING A WINNER AND ANALYTICS
When running an automated campaign and the campaign has run long enough to achieve statistical significance, the next step is to select a winner for the campaign. By selecting a winner, the remaining recipients will receive the email campaign that performed better. Select a winner by following the below steps.
Navigate to the active campaign with the A/B test.
Click the A/B Test or Auto-Winner A/B Test tab at the top of the campaign.
A/B TEST
Navigate to the bottom of the page.
Click the Select as winner button on the applicable variant.
By selecting a winner, this campaign will turn into a single-content version, sending the version that has been selected as the winner. The A/B versions will no longer be available.
AUTO-WINNER A/B TEST
A winner is automatically selected with this test type. Once a winner is determined, the winning variant is sent to the audience determined in the holdout percentage. View a complete test summary with test details, how each variant performed, which variant was the winner, and the results of the holdout group send.
REVIEW PAST A/B TEST ANALYTICS
To review analytics or results from a campaign's previous A/B Tests, open the campaign. Then, on the A/B Test tab, click the link in the top right corner labeled View Past Tests.
Then, you should see a page that is labeled Past Tests with previous variant metrics.
