Holdout Groups are a group of users purposely separated from the rest of an audience, used to test against when sending an email campaign. Separating this audience with a Holdout Group against a broader audience helps prove incremental lift of an individual campaign.
Holdout groups can be used on any automated or one-time campaign (including triggers and smart campaigns). Holdout groups perform best on campaigns that have a high send volume, ensuring statistical significance is achieved between the control group and the test group quickly.
How Holdout Groups are determined
How Holdout Groups are determined
Holdout Groups are generated by a hashing function that’s used to create two groups: a control and a test group. Each campaign running a holdout is given a unique and random string, which is appended to each email address that qualifies for the audience. Since the string is generated for each campaign and is unique and random per campaign, it ensures the group selected for the holdout in a campaign is random. This hashing function converts the string of email address and unique random string for the campaign into an integer for each audience member, which is then used to compare to the holdout percentage assigned to that campaign.
Setting up a Holdout Group
To set up an Holdout Group, build your campaign normally, following instructions in the Campaign Builder guide.
When you get to the Settings Tab, you have the option of including a Holdout Group.
NOTE: If you’re configuring an audience with a frequency cap, the frequency cap is honored prior to the holdout percentage being applied.
Below is an overview of the options when setting up a Holdout Group:
Holdout percentage
Required field
The percentage of the audience that will not see the campaign. Bluecore recommends between 5-10% of your audience in the holdout group.
Winning criteria
Optional field
The metric that determines a successful test. Input the percentage lift for the campaign when compared to the Holdout Group, and what metric is being used to test.
Conversion Rate
The number of customers who purchased, divided by the number of customers who received an email.
Revenue per Buyer
Average revenue a customer contributes during the test period
Revenue per Order
Average revenue an individual order contributes during the test period
Description
Optional field
Use this field to add any additional information about your test parameters, such as the reason for the current test, expected results, or any other relevant considerations.
Analyzing your Holdout Groups
The best way to analyze your Holdout Groups is through the Experimentation Hub, which allows you to review and manage your campaigns’ Holdout Groups in one centralized location within Bluecore.
Experimentation Hub optimizes your time spent on testing by letting you know when a test is ready to declare a winner.
For more information on how to get analytics around Holdout Groups, see Experimentation Hub.