The Experimentation Hub allows you to review and manage your campaigns’ A/B tests and Holdout Groups in one centralized location within Bluecore. This will save you time from needing to go into each campaign to review the A/B Test or Holdout data. Now, you can see every campaign with an active, paused, or completed test and view its latest results in the Experimentation Hub.
Experimentation Hub optimizes your time spent on testing by letting you know when a test is ready to declare a winner. It will save you time calculating results - you don’t have to worry about calculating whether a result is statistically significant anymore.
ℹ️ Your experiments will still need to be set up individually at the campaign level.
A/B Tests
What is an A/B Test?
What is an A/B Test?
A/B Tests allow you to test different elements of a campaign to make data-driven decisions about which aspects of a campaign are performing best. Use A/B Tests to compare two or more subject lines, product recommendations, offers, or more.
This is a look at the Experimentation Hub:
Filters
There are filters at the top of the dashboard you can use to adjust which tests to view.
Active Status
Filters the A/B Tests into three possible categories:
Active - Tests that are currently running
Complete - Tests that have finished and have results that can be viewed, with a full backlog of all previous tests
Paused - Tests that aren’t yet complete, but have been paused
Analysis Metric
The currently selected metric for the analysis of this test:
Conversion Rate - The number of customers who purchased, divided by the number of customers who received an email.
Unique CTR - The number of customers who clicked, divided by number of customers who received an email.
Significance Threshold (P value)
Adjust the level of confidence of the result of the test. A lower P value means more confidence in the result of the test. For example, a P value of 0.02 means there’s a 2% chance of the result occurring due to random chance alone, instead of via customer actions.
Is Significant?
Filtering just the statistically significant tests will only show tests that meet the significance threshold in the previous filter. For example, a test with a 0.15 P value would be considered significant if the significance threshold is 0.2, but not if the significance threshold is 0.1.
Channel
Email or SMS
Campaign
Narrow the dashboard to show data for only specific campaigns.
Journey
Narrow the dashboard to show data for only specific Journeys.
The percentage of A/B Tests currently achieving statistical significance, based on the filters above. Narrowing the selection of tests will change this percentage.
Number of tests ready for action, based on the filters above. Narrowing the selection of tests will change this number.
In the Experimentation Hub, “ready for action” means one of two things:
At least one variant in the test has reached a statistically significant result
The test has been active for at least 30 days, and you should assess whether the test should continue
Under the Ready for Action list, you can view campaigns that need attention and meet one of the above criteria.
All Tests View
Under All Tests, you can see all tests, see how long they’ve been active, whether they’ve reached statistical significance, and the latest results of the tests.
Campaign Name
The name of the campaign in Bluecore.
Start Date
Date the A/B Test began.
Name
The name of the A/B Test.
Type
The type of test chosen when setting up the campaign
Incentives
Recommendations
Creative
Subject Line/Copy
Personalization
Custom
Test Metric
Winning criteria selected when setting up the campaign
Conversion Rate
Click Through Rate
Description
Additional information added when setting up the campaign
State
Displays if a campaign is Active, Paused, or Complete.
Variant Allocation
Percentage of the delivered emails are the test variant.
Days Active
Days the A/B Test has been/was active.
Delivered
Test variant email delivered by the campaign.
Total Delivered
Total emails delivered by the campaign across all variants.
Analysis Status
Displays the latest status of the test, such as “Significance reached,” or if it still may need more time.
Analysis Metric
Determined by one of the filters chosen at the top of the page.
Variant ID
Displays “Variant B.”
Variant Name:
Displays the name of the test variant entered in Bluecore when setting up the A/B test.
Test Variant Performance
Performance of the test variant, based on the Success Metric chosen.
Variant A Performance
Performance of Variant A, based on the Success Metric chosen.
Difference vs Variant A
How much better or worse the test variant is than Variant A, as a percentage. Green is positive, red is negative. A green result means the test variant performed better than Variant A.
Is Significant?
Displays whether the result of the test is statistically significant.
P value
The confidence level of the result, as expressed as a decimal. A lower P value means more confidence in the result of the test.
Action
Any potential actions will appear here as links. Clicking on “Review Results & Select Winner” will redirect you to the Bluecore campaign page, where you can see the same results and choose a winner of the A/B Test.
There are two insights available to learn more about the current tests - A/B tests by month, and by active days. These can help demonstrate how many A/B tests have been historically run and for how long.
To get the best use out of the Experimentation Hub, check it about once a week to see if any new tests have reached statistical significance.
Test Variants
In Experimentation Hub, Variant B is always shown as the test, no matter what you call your variants when setting up your A/B Test. We recommend making Variant A your control and using Variant B (and/or variants C, D, and E) as the test.
This way, you can see how the test variant performed against the control in the Difference vs Variant A column - a green, positive number means Variant B (the test variant) performed better than Variant A. A red, negative number indicates Variant A performed better than Variant B.
Holdout Groups
Holdout Groups allow you to keep a percentage of eligible recipients from seeing your campaign, letting you know how effective the campaign is when compared to not seeing it.
This is a look at the Experimentation Hub:
Filters
There are filters at the top of the dashboard you can use to adjust which tests to view.
Holdout Group State
Filters the tests into two categories:
Active
Tests that are currently running
Complete
Tests that have finished and have results that can be viewed, with a full backlog of all previous tests
Analysis Metric
The currently selected metric for the analysis of this test:
AOV Per Customer
Tracks the average order value of customers in the test group versus those in the holdout group
Conversion Rate
Tracks the percentage of customers who made a purchase in the test group compared to the holdout group
Cross Sell Conversion Rate
Tracks the percentage of customers in the test group who purchased additional products in other categories versus those in the holdout group
Number of Orders per Customer
Tracks the number of orders per customer in the test group versus the holdout group
Revenue per Customer
Tracks the average revenue per customer in the test group versus the holdout group
Significance Threshold (P value)
This will adjust the level of confidence of the result of the test. A lower P value means more confidence in the result of the test. For example, a P value of 0.02 means there’s a 2% chance of the result occurring due to random chance alone, instead of via customer actions.
Is Significant?
Filtering just the statistically significant tests will only show tests that meet the significance threshold in the previous filter. For example, a test with a 0.15 P value would be considered significant if the significance threshold is 0.2, but not if the significance threshold is 0.1.
Channel
Either Email or SMS, depending on if you want to specify a channel.
Campaign
Narrow the dashboard to show data for only specific campaigns
Journey
Narrow the dashboard to show data for only specific Journeys
The percentage of Holdout Group tests currently achieving statistical significance, based on the filters above. Narrowing the selection of tests will change this percentage.
Number of tests ready for action, based on the filters above.
In the Experimentation Hub, “ready for action” means one of two things:
At least one variant in the test has reached a statistically significant result
The test has been active for at least 90 days, and you should assess whether the test should continue. This is intentionally different from the A/B Tests using 30 days as a benchmark.
30 Days vs 90 Days
With Holdout Groups, more data is needed because it tracks the purchase behavior of the same customers over an extended period, requiring a larger sample size to account for factors like dropouts or the time needed to see some customers go through more than one buying cycle.
In contrast, A/B Tests compare outcomes between two groups at a single point in time, which typically needs less data to achieve statistically significant results.
Under the Ready for Action list, you can view campaigns that need attention and meet one of the above criteria.
All Tests View
Under All Tests, you can see all tests, see how long they’ve been active, whether they’ve reached statistical significance, and the latest results of the tests.
Journey/Campaign
Name of the Journey or campaign in Bluecore.
Start Date
Date the Holdout Group test began.
End Date
Date the Holdout Group test ended.
Test Metric
Winning criteria selected when setting up the campaign.
Conversion Rate
Revenue per Buyer
Revenue per Order
Expected Lift
Percentage lift for the campaign when compared to the Holdout Group, selected when setting up the campaign.
Description
Additional information added when setting up the campaign
Holdout Allocation
What percentage of the campaign emails are/were being held back from delivery.
State
Whether the campaign is Active, Paused, or Complete.
Days Active
Days the Holdout Group test has been active.
% of Min. Sample Size
How close the campaign is to reaching a minimum sample size for statistical significance. Some of the factors here include total participants and holdout allocation.
Participants
Customers who did not get emails who otherwise would’ve.
Analysis Status
Either “Minimum sample size reached,” “Running for 90+ days,” or “Running for 180+ days” depending on if the test has actionable results.
Analysis Metric
Determined by one of the filters chosen at the top of the page.
Test Group Performance
Performance of customers who received the campaign, based on the Success Metric chosen.
Holdout Group Performance
Performance of customers who didn’t receive the campaign, based on the Success Metric chosen.
Difference vs Holdout
How much better or worse the test group is vs the Holdout Group, as a percentage. Green is positive, red is negative. A green result means the test group performed better than the Holdout Group.
Is Significant?
Displays whether the result of the test is statistically significant.
P value
The confidence level of the result, as expressed as a decimal. A lower P value means more confidence in the result of the test.
Action
Any potential actions will appear here as links. Clicking on any links here will redirect you to the Bluecore campaign page, where you can see the same results and choose to continue the Holdout Group.
There are two insights available to learn more about the current tests - Holdout Group tests by month, and by active days. These can help demonstrate how many Holdout Group tests have been historically run and for how long.
To get the best use out of the Experimentation Hub, check it about once a week to see if any new tests have reached statistical significance.