What is a Holdout Group?
A holdout group is a segment of your audience deliberately excluded from a marketing campaign or channel activation. This group serves as a control, allowing you to measure the genuine uplift caused by your marketing efforts rather than attributing all conversions to your campaign.
Why Holdout Groups Matter
Traditional attribution models assume that everyone who converts after seeing an ad was influenced by it. This creates inflated ROI claims. A holdout group reveals how many conversions would have happened anyway – what marketers call "organic" or "baseline" conversion rates.
Without holdout groups, you risk:
- Overestimating campaign impact by crediting natural customer behaviour to paid media
- Misallocating budget to underperforming channels
- Making poor strategic decisions based on false attribution data
For UK agencies managing multi-channel campaigns across competitive sectors (FMCG, retail, financial services), holdout groups provide the ground truth needed for confident optimisation.
How They Work
You randomly select a statistically significant portion of your audience – typically 5-10% – and exclude them from the campaign entirely. The remaining 90-95% receives your ads across chosen channels (search, display, social, TV, etc.).
After the campaign runs, you compare:
- Holdout group conversion rate (no exposure)
- Exposed group conversion rate (with exposure)
The difference is your true incremental lift. If your holdout group converts at 2% and exposed group at 3%, your incremental uplift is 1 percentage point.
When to Use Holdout Groups
Best for: - Large-scale campaigns with substantial budgets (the forgone revenue from 5-10% of audience is acceptable) - Brand awareness or consideration campaigns where measurement is ambiguous - Testing new channels or strategies - Mature campaigns where baseline conversion patterns are established
Less practical for: - Performance campaigns with limited budget (the cost of excluding potential converters is high) - Short-duration campaigns (insufficient time to gather statistical significance) - Niche audiences where 10% exclusion is prohibitively expensive
UK Media Context
Holdout groups are increasingly standard practice among sophisticated UK agencies and brands, particularly post-iOS privacy changes. As third-party cookies deprecate, holdout experiments offer a privacy-compliant alternative to cross-device tracking for measuring incrementality.
The Financial Conduct Authority and ASA expect higher evidentiary standards for marketing claims, making holdout-backed ROI figures valuable for compliance and stakeholder confidence.
Best Practices
- Ensure holdout groups are randomly selected to avoid bias
- Maintain sufficient sample size for statistical validity
- Isolate variables – only exclude from one channel, not all marketing
- Run for long enough to capture full customer journey
- Document assumptions clearly for stakeholder transparency