Holdout groups are a group of users purposely separated from the rest of an audience, used to test against when sending an email campaign. Separating this audience with a holdout group against a broader audience helps prove incremental lift of an individual campaign.
Holdout groups can be used on any automated or one-time campaign (including triggers and smart campaigns). Holdout groups perform best on campaigns that have a high send volume, ensuring statistical significance is achieved between the control group and the test group quickly.
GETTING STARTED
When setting up a holdout group, please reach out to your CSM to begin the process of holdout groups reporting via a file transfer to the SFTP.
SETTING UP AN AUDIENCE
The first step to configuring a holdout group is to determine an audience for the campaign to be sent to. Audiences for holdout groups can be configured using Bluecore’s audience builder and selected when building a campaign.
Click here to learn more about creating an audience and adding it to a campaign.
NOTE: If you’re configuring an audience with a frequency cap, the frequency cap is honored prior to the holdout percentage being applied.
ADDING A HOLDOUT GROUP
After an audience has been created, navigate to the campaign settings of a new campaign or existing campaign to add a holdout group.
NOTE: To add a holdout group to an existing campaign, please create a draft of the campaign, follow the below instructions to add a holdout group, and republish the campaign.
When adding a holdout group, enter a percentage of the audience to be held out.
TIP: Bluecore recommends using a 5-10% of your audience in the holdout group.
STOPPING A HOLDOUT TEST
Holdout tests run on campaigns until the campaign is completed or the holdout group is turned off.
TIP: Bluecore recommends running a holdout test for a duration of three months. This time frame allows statistical significance to be achieved, but may vary based on audience size.
Stop a holdout test by following the below steps.
Navigate to a campaign that’s running the test.
Click edit on the campaign.
Navigate to the holdout groups section of campaign settings.
Select No to not include a holdout group for this campaign.
Republish the draft.
Be sure to repeat the above steps to stop any holdout tests for a current campaign.
ANALYZING YOUR HOLDOUT GROUPS
Holdout testing is useful for understanding how an individual campaign performs. The goal of analysis is to determine the purchase behavior of customers in the holdout group versus those who received the campaign to understand the incremental lift of the campaign.
Bluecore will provide a standard report in the form of a .csv file containing the raw data for your team’s analysis on a weekly basis once initially requested. The report will contain the following information for all holdout tests you're currently running:
Email: The email address of the customer.
Date: The timestamp of when the customer was delivered the email or was held out of the campaign send.
Campaign Name: The name of the campaign running the holdout.
Campaign ID: The ID of the campaign running the holdout.
Holdout (T/F): A boolean indicator showing if the customer was in the holdout group (true) or was delivered an email (false).
NOTE: If you require a customized report outside of the standard report, this may result in an additional technical service cost. Please contact your CSM for additional details.
FAQ
Q: Do I need to change my existing workflow if I’m already running a holdout test?
A: Bluecore recommends following the above implementation as your current workflow.
Q: How does this feature change my workflow over time?
A: Holdout groups reduce the amount of time spent to set up and configure a holdout test. Simply select to do a holdout with a click of a button in your campaign settings.
Q: Can I use a holdout group with a multi-touch campaign?
A: Use a holdout group with Bluecore’s Experience Designer. The holdout group will be held out of the first touch, and therefore not be able to qualify for any touch afterwards.
Q: Can I adjust the holdout percentage after the campaign has launched?
A: Start by creating a new draft of the campaign. Then, go to the Campaign Settings, and adjust the percentage you’d like to holdout. Please note that this will effectively reset the holdout data, and you’ll need to restart your analytics from the time the campaign has been republished.
Q: How do holdout groups work with audience-level frequency caps?
A: Audience-level frequency caps are honored prior to the holdout percentage being applied.
Q: Can I run a holdout test on any campaign?
A: Holdout testing is available for all campaigns except for transactional and any campaign using send time optimization.
Q: How does the minimum time between emails (MTBE) affect a holdout group?
A: MTBE will directly impact the size of your control group vs. test group. Ideally, if you put any additional restrictions or internal dependencies on the email, these should be as short as possible as it may be the case that someone ends up in the control group without honoring the MTBE or the internal dependency.
Q: Why is the percentage of held out customers growing over the lifetime of the campaign?
A: The percentage of held out customers may be increasing based on the initial configuration of the audience. Audiences with a large enough lookback window may allow customers to qualify for more than one campaign with the same event as the audience is growing.
These audiences strategies do not result in bias of hold out selection. The count of unique email addresses over a certain period of time will reflect the desired holdout group percentage. Replenishment campaigns and audiences commonly exhibit this behavior.
Q: Do frequency caps affect the size of an audience or holdout group?
A: Audiences that target customers who didn’t receive an email and use frequency capping will continue to pull held out customers into that audience. This group of customers will always be held out. The non-held out group of customers that did receive an email will not reappear in subsequent sends that violate audience behavior configuration or frequency capping rules.
Q: How are control groups determined?
A: Holdout groups are generated by a hashing function that’s used to create two groups: a control and a test group. Each campaign running a holdout is given a unique and random string, which is appended to each email address that qualifies for the audience. Since the string is generated for each campaign and is unique and random per campaign, it ensures the group selected for the holdout in a campaign is random. This hashing function converts the string of email address and unique random string for the campaign into an integer for each audience member, which is then used to compare to the holdout percentage assigned to that campaign.