With Bluecore Communicate Email, you can set up A/B Test campaigns to send multiple variant options to your shopper segments. This helps you prove incremental lift and determine which variant is the most performant, so you can continue sending THAT version to the remainder of the population!
However, you have a couple of methods to choose a winning variant:
You can manually review A/B test results to choose a winner, OR
You can utilize Bluecore's auto-winner feature, which automatically takes the winning variant with statistical significance.
Not sure which option is best for you? Let's go over the details of each!
MANUAL REVIEW
Manually reviewing A/B Test results is available for both one-time and automated campaigns.
In one-time sends, you must wait until the campaign has finished deploying to view real results. Because of this, you cannot impact that same send since it's already over.
You cannot interrupt a one-time send if you start to see a clear winner immediately.
There's no statistical significance monitoring; it's up to you to monitor!
You have a larger sample size to view results from since the variants are going through the entire campaign.
AUTO-WINNER
Auto-winner is only available for one-time campaigns.
You get to let your insights impact the same campaign you're testing, since the holdout group on that campaign gets the winning version. This is inherently different from a typical A/B Test where the opportunity to impact that tested campaign doesn't exist.
You must commit to your winning KPI in advance since the structure depends on hitting the KPI you choose.
NOTE: Retailers should be doing this anyway. 😀
Your sample size is smaller on the initial variants, because a larger portion of the audience is being saved for the holdout group.
You have the opportunity to allow Bluecore to "prematurely" declare a winner before the test duration has completed, if we can determine a winner with 95% confidence before then.
If you select the longest possible test length (10 days), you have to wait the full 10 days for the test to finish before the holdout group receives the winning version.
NOTE: If you have a campaign that's time sensitive and you don't want to wait 10 days to determine a winning variant, be sure to choose a shorter test duration.