Understand holdout groups

Holdout groups measure performance for new campaigns by only sending the campaign to a certain percentage of the selected audience. 

By comparing the behavior of customers who received the campaign to customers who did not, you can understand the campaign’s impact. The difference in performance between these two groups is called lift.

Holdout group percentages apply to all customers who qualify over the campaign’s lifetime, not to individual campaign sends.

To learn how to add a holdout group to a campaign, see the following articles:

• Holdout group best practices
• Add a holdout group to an email campaign
• Add a holdout group to an SMS/MMS campaign

How holdout groups work

Holdout groups randomly assign customers into a test group (receives the campaign) and control group (does not receive the campaign).

When a holdout campaign sends for the first time, Bluecore applies the frequency cap first, then randomly divides the audience into the test and control groups based on the holdout percentage. 

Once assigned to a test or control group, the customer’s group assignment never changes for that campaign.

When a recurring campaign runs again, requalifying customers keep their original group assignment. New qualifying customers are randomly assigned to either the test or control group, using the holdout percentage.

In long-running automated campaigns, test and control group sizes for each send may shift over time as customers requalify for or leave the audience. This shift is expected.

For example, if a campaign has 100 qualifying customers and a 50% holdout group, Bluecore assigns 50 customers to the control group and 50 customers to the test group.

When the campaign runs the next week, the following customers qualify:

• 25 from the control group
• 50 from the test group
• 100 new customers

The 75 returning customers (25 from the control group and 50 from the test group) keep their group association. The 100 new customers are split 50/50 between test and control groups.

Now for the second send, the audience is split in this way:

• 75 in the control group
• 100 in the test group

While the second send seems to target the customers in a 42/58 split, the campaign still maintains a 50/50 split across all customers who qualified so far.

Holdout percentage

The holdout percentage defines the percentage of the audience that will be in the control group.

Bluecore generally recommends a five to 10% holdout percentage. 

Test metrics

Test metrics are optional metrics you can select to measure the success of your holdout group. If selected, test metrics are tracked in the Experimentation Hub.

You can choose from the following test metrics: