Holdout group best practices

You can add holdout groups to any email or SMS/MMS campaign, but Bluecore recommends the following best practices for when to add a holdout group to a campaign.

To learn more about holdout groups, see the following articles:

You have a hypothesis you want to test

Holdout groups work best when you have a hypothesis or goal that you want to test in advance of creating the campaign.

First, define what you want to measure before creating the campaign.

For example, if you want to drive incremental revenue, a holdout group can measure the increase by comparing behavior of customers who received the campaign and those who didn’t.

If your goal is broad, such as increasing overall revenue, don’t use a holdout group. Holdout groups should test a specific hypothesis and, because they exclude a percentage of customers, they can also reduce the campaign’s revenue potential.

Remember that hypotheses should be unbiased, and the results of the tests are based on your customer’s data.

If the results don’t support your hypothesis, you can stop the test and accept the results or create a new campaign.

You are creating a new experience

If you are creating a new campaign, adding a new audience segmentation to an existing campaign, or creating another new kind of promotional experience for your customers, a holdout group can measure whether it performs as expected.

If you need to add a new experience regardless of the lift, such as adding a missing campaign type, a holdout group isn’t helpful.

Holdout groups are only useful when you want to test a new experience, not for changes you plan to make regardless of the results.

The campaign has high send volume

Adding a holdout group to a high volume campaign makes it more likely that you will get statistical significance, which measures the likelihood that the difference in data was caused by a true effect as opposed to random chance.

Higher volume campaigns typically reach statistical significance faster because they include more customers.

Lower volume campaigns may take longer to reach statistical significance because they don’t reach as many customers.

There is an end date

Holdout group tests should have an end date and not run indefinitely.

Use the Experimentation Hub to track when the campaign has reached statistical significance.

The campaign should stay the same the entire duration

Don’t change a campaign with a holdout group until the test is complete.

Changing the control group size, campaign content, or audience criteria can invalidate the test.

If you need to change the campaign, pause the existing campaign and create a new one.