The Experimentation Hub allows you to review and manage your campaigns’ A/B tests and Holdout Groups in one centralized location within Bluecore. This will save you time from needing to go into each campaign to review the A/B Test or Holdout data. Now, you can see every campaign with an active, paused, or completed test and view its latest results in the Experimentation Hub.
Experimentation Hub optimizes your time spent on testing by letting you know when a test is ready to declare a winner. It will save you time calculating results - you don’t have to worry about calculating whether a result is statistically significant anymore.
ℹ️ Your experiments will still need to be set up individually at the campaign level.
A/B Tests
What is an A/B Test?
What is an A/B Test?
A/B Tests allow you to test different elements of a campaign to make data-driven decisions about which aspects of a campaign are performing best. Use A/B Tests to compare two or more subject lines, product recommendations, offers, or more.
This is a look at the Experimentation Hub:
Filters
There are filters at the top of the dashboard you can use to adjust which tests to view.
Active Status: Filters the A/B Tests into three possible categories:
Active - Tests that are currently running
Complete - Tests that have finished and have results that can be viewed, with a full backlog of all previous tests
Paused - Tests that aren’t yet complete, but have been paused
Success Metric: These are the two primary metrics Bluecore uses for its analysis:
Conversion Rate - The number of customers who purchased, divided by the number of customers who received an email.
Unique Click Through Rate - The number of customers who clicked, divided by number of customers who received an email.
Significance Threshold (P value): Adjust the level of confidence of the result of the test. A lower P value means more confidence in the result of the test. For example, a P value of 0.02 means there’s a 2% chance of the result occurring due to random chance alone, instead of via customer actions.
Is Significant?: Filtering just the statistically significant tests will only show tests that meet the significance threshold in the previous filter. For example, a test with a 0.15 P value would be considered significant if the significance threshold is 0.2, but not if the significance threshold is 0.1.
Channel - This is Email, but could be expanded in future versions for additional channels.
Campaign: Narrow the dashboard to show data for only specific campaigns.
Journey: Narrow the dashboard to show data for only specific Journeys.
The percentage of A/B Tests currently achieving statistical significance, based on the filters above. Narrowing the selection of tests will change this percentage.
Number of tests ready for action, based on the filters above. Narrowing the selection of tests will change this number.
In the Experimentation Hub, “ready for action” means one of two things:
At least one variant in the test has reached a statistically significant result
The test has been active for at least 30 days, and you should assess whether the test should continue
Under the Ready for Action list, you can view campaigns that need attention and meet one of the above criteria.
All Tests View
Under All Tests, you can see all tests, see how long they’ve been active, whether they’ve reached statistical significance, and the latest results of the tests.
Campaign Name: The name of the campaign in Bluecore.
Start Date: Date the A/B Test began.
A/B Test Status: Displays if a campaign is Active, Paused, or Complete.
Variant Allocation: Percentage of the delivered emails are the test variant.
Days Active: Days the A/B Test has been/was active.
Delivered: Test variant email delivered by the campaign.
Total Delivered: Total emails delivered by the campaign across all variants.
Analysis Status: Displays the latest status of the test, such as “Significance reached,” or if it still may need more time.
Success Metric: Determined by one of the filters chosen at the top of the page.
Variant: Displays the name of the test variant entered in Bluecore when setting up the A/B test.
Test Variant Performance: Performance of the test variant, based on the Success Metric chosen.
Variant A Performance: Performance of Variant A, based on the Success Metric chosen.
Difference vs Variant A: How much better or worse the test variant is than Variant A, as a percentage. Green is positive, red is negative. A green result means the test variant performed better than Variant A.
Is Significant?: Displays whether the result of the test is statistically significant.
P value: The confidence level of the result, as expressed as a decimal. A lower P value means more confidence in the result of the test.
Action: Any potential actions will appear here as links. Clicking on “Review Results & Select Winner” will redirect you to the Bluecore campaign page, where you can see the same results and choose a winner of the A/B Test.
There are two insights available to learn more about the current tests - A/B tests by month, and by active days. These can help demonstrate how many A/B tests have been historically run and for how long.
To get the best use out of the Experimentation Hub, check it about once a week to see if any new tests have reached statistical significance.
Test Variants
In Experimentation Hub, Variant B is always shown as the test, no matter what you call your variants when setting up your A/B Test. We recommend making Variant A your control and using Variant B (and/or variants C, D, and E) as the test.
This way, you can see how the test variant performed against the control in the Difference vs Variant A column - a green, positive number means Variant B (the test variant) performed better than Variant A. A red, negative number indicates Variant A performed better than Variant B.
Holdout Groups
What is a Holdout Group?
Holdout Groups allow you to keep a percentage of eligible recipients from seeing your campaign, letting you know how effective the campaign is when compared to not seeing it.
This is a look at the Experimentation Hub:
Filters
There are filters at the top of the dashboard you can use to adjust which tests to view.
Holdout Group State: Filters the tests into two categories:
Active - Tests that are currently running
Complete - Tests that have finished and have results that can be viewed, with a full backlog of all previous tests
Success Metric: The primary metrics Bluecore uses for its analysis of Holdout Groups:
AOV Per Customer - Tracks the average order value of customers in the test group versus those in the holdout group
Conversion Rate - Tracks the percentage of customers who made a purchase in the test group compared to the holdout group
Cross Sell Conversion Rate - Tracks the percentage of customers in the test group who purchased additional products in other categories versus those in the holdout group
Number of Orders per Customer - Tracks the number of orders per customer in the test group versus the holdout group
Revenue per Customer - Tracks the average revenue per customer in the test group versus the holdout group
Significance Threshold (P value): This will adjust the level of confidence of the result of the test. A lower P value means more confidence in the result of the test. For example, a P value of 0.02 means there’s a 2% chance of the result occurring due to random chance alone, instead of via customer actions.
Is Significant?: Filtering just the statistically significant tests will only show tests that meet the significance threshold in the previous filter. For example, a test with a 0.15 P value would be considered significant if the significance threshold is 0.2, but not if the significance threshold is 0.1.
Channel - Either Email or SMS, depending on if you want to specify a channel.
Campaign: Narrow the dashboard to show data for only specific campaigns
Journey: Narrow the dashboard to show data for only specific Journeys
The percentage of Holdout Group tests currently achieving statistical significance, based on the filters above. Narrowing the selection of tests will change this percentage.
Number of tests ready for action, based on the filters above.
Narrowing the selection of tests will change this number.In the Experimentation Hub, “ready for action” means one of two things:
At least one variant in the test has reached a statistically significant result
The test has been active for at least 90 days, and you should assess whether the test should continue. This is intentionally different from the A/B Tests using 30 days as a benchmark.
30 Days vs 90 Days
With Holdout Groups, more data is needed because it tracks the purchase behavior of the same customers over an extended period, requiring a larger sample size to account for factors like dropouts or the time needed to see some customers go through more than one buying cycle.
In contrast, A/B Tests compare outcomes between two groups at a single point in time, which typically needs less data to achieve statistically significant results.
Under the Ready for Action list, you can view campaigns that need attention and meet one of the above criteria.
All Tests View
Under All Tests, you can see all tests, see how long they’ve been active, whether they’ve reached statistical significance, and the latest results of the tests.
Journey/Campaign: Name of the Journey or campaign in Bluecore.
Start Date: Date the Holdout Group test began.
Holdout Allocation: What percentage of the campaign emails are/were being held back from delivery.
Holdout State: Whether the campaign is Active, Paused, or Complete.
Days Active: Days the Holdout Group test has been active.
% of Min. Sample Size: How close the campaign is to reaching a minimum sample size for statistical significance. Some of the factors here include total participants and holdout allocation.
Participants: Customers who did not get emails who otherwise would’ve.
Analysis Status: Either “Minimum sample size reached,” “Running for 90+ days,” or “Running for 180+ days” depending on if the test has actionable results.
Success Metric: Determined by one of the filters chosen at the top of the page.
Test Group Performance: Performance of customers who received the campaign, based on the Success Metric chosen.
Holdout Group Performance: Performance of customers who didn’t receive the campaign, based on the Success Metric chosen.
Difference vs Holdout: How much better or worse the test group is vs the Holdout Group, as a percentage. Green is positive, red is negative. A green result means the test group performed better than the Holdout Group.
Is Significant?: Displays whether the result of the test is statistically significant.
P value: The confidence level of the result, as expressed as a decimal. A lower P value means more confidence in the result of the test.
Action: Any potential actions will appear here as links. Clicking on any links here will redirect you to the Bluecore campaign page, where you can see the same results and choose to continue the Holdout Group.
There are two insights available to learn more about the current tests - Holdout Group tests by month, and by active days. These can help demonstrate how many Holdout Group tests have been historically run and for how long.
To get the best use out of the Experimentation Hub, check it about once a week to see if any new tests have reached statistical significance.