Incrementality testing measures the conversions or revenue a marketing activity caused beyond what would have happened anyway. It compares a treatment group that receives the activity with a control group that does not. The difference is incremental lift.
If 720 people in a campaign's treatment group purchase, that is not proof the campaign caused all 720 purchases. A control group can estimate how many would have happened without it. That estimate is what makes an incrementality test useful for budget decisions.
How does incrementality testing work?
- Choose one decision and one primary outcome. For example, decide whether to keep funding a retargeting campaign, measured by completed purchases.
- Assign comparable groups before the campaign runs. Randomize eligible users where possible. For a geo experiment, match markets using pre-test sales and demand patterns.
- Change only the tested activity. The treatment group receives it; the control group does not. Keep other campaigns, prices, and promotions as consistent as possible.
- Measure outcomes in both groups over the same period. Include the normal time between exposure and conversion.
- Compare rates, uncertainty, and cost. Calculate lift, check whether the estimate is precise enough to guide the decision, and compare incremental revenue with campaign spend.
The test estimates the effect of the specific campaign, audience, and period you tested. It does not prove that every future campaign will have the same lift.
How to calculate incremental lift
Compare rates, not raw conversion counts, when the groups have different sizes. Consider this user holdout test:
| Metric | Treatment | Control |
|---|---|---|
| Eligible people | 12,000 | 10,000 |
| Purchases | 720 | 500 |
| Purchase rate | 6% | 5% |
The control rate predicts 12,000 × 5% = 600 purchases in a treatment-sized group without the ads. The treatment produced 720, so the estimated result is 120 incremental purchases.
| Measure | Formula | Result |
|---|---|---|
| Absolute lift | Treatment rate − control rate | 1 percentage point |
| Relative incremental lift | (Treatment rate − control rate) ÷ control rate | 20% |
| Incremental conversions | Treatment conversions − (control rate × treatment group size) | 120 |
| Incremental share of treatment conversions | Incremental conversions ÷ treatment conversions | 16.7% |
Lift vs. incrementality: The 20% figure is relative lift against the control baseline. The 16.7% figure is the share of treatment purchases estimated to be incremental. Both describe the same experiment with different denominators. Label the metric and show the formula when reporting a result.
For revenue, multiply incremental purchases by their average order value only if that value is comparable across groups. At $80 per incremental purchase, this example produces an estimated $9,600 in incremental revenue. If the tested campaign cost $4,000, incremental ROAS (iROAS) = $9,600 ÷ $4,000 = 2.4. Use measured revenue by group when order values differ.
Choose the right incrementality test
| Test design | Use it when | Main check |
|---|---|---|
| User holdout | You can randomly withhold an ad or message from eligible people | Prevent control users from receiving the tested activity elsewhere |
| Geo experiment | You cannot assign users individually, such as for out-of-home or broad regional media | Match markets before the test and watch for local changes |
| Platform conversion lift study | An ad platform supports a managed treatment and holdout study | Confirm which campaigns and conversion events the platform includes |
User holdout tests
Randomly assign eligible users to treatment and control groups. Show the tested ads or messages only to treatment users. This is practical for lifecycle email, retargeting, and supported ad-platform lift studies. Keep assignment stable, deduplicate people across devices where possible, and check whether control users can still see the same campaign through another channel.
Geo experiments
Select regions with similar pre-test outcomes. Run the tested campaign in treatment markets and withhold it from control markets, or increase spend only in treatment markets. Compare the change in outcomes after accounting for their different baselines. A simple post-test sales difference is unreliable when markets already had different demand. Geo tests help with channels where user-level holdouts are unavailable, but local promotions, weather, inventory, and spillover across borders can distort the result.
Platform conversion lift studies
Ad platforms can manage audience assignment and report incremental conversions. Check eligibility, holdout size, conversion coverage, and which campaigns the study includes before launch. A platform study measures the activity and matched outcomes it can observe; it is not a complete view of every marketing channel. Google Ads documents how its user-based Conversion Lift studies set up holdouts and assess study power.
Plan the test before you spend
Write down the decision rule before seeing results. A useful plan records:
- The campaign, eligible audience, and exact treatment being tested
- One primary conversion or revenue outcome, plus any secondary outcomes
- The expected baseline rate and smallest lift that would change the budget decision
- Group assignment, sample size, test length, and conversion lag
- Campaigns, promotions, inventory changes, and other possible interference
- How you will handle an inconclusive or negative result
Run a power analysis with the baseline rate and smallest useful lift. It tells you whether the available audience and duration can detect a result of that size. A small holdout may reduce the cost of withholding ads but require a longer study. If conversion volume is too low, extend the test or test a larger eligible audience. Do not call a noisy result "no lift" simply because it failed a significance threshold.
Once the test ends, report the treatment and control rates, incremental conversions, relative lift, and an uncertainty interval. If the interval includes both a meaningful loss and a meaningful gain, the test did not settle the budget question. Avoid stopping early just because a live dashboard briefly shows a favorable number.
Measure the same outcome in both groups
Random assignment cannot fix missing or duplicated conversions. Check that the purchase or lead event fires once, uses the same definition in both groups, and arrives within the analysis window. Apply the same consent rules and identity handling to treatment and control. For offline sales or qualified leads, join outcomes back to assigned groups using permitted first-party identifiers.
Hardal's first-party server-side collection can capture and route permitted conversion events before they reach ad platforms or measurement tools. It does not create a holdout or prove causality on its own. It gives an experiment a more consistent outcome feed. After moving critical events to server-side collection, arabam.com measured 80.1% more PageView events and 22.8% more purchase conversions than with its earlier browser-only setup. That case shows why event coverage should be checked before interpreting lift.
Incrementality vs. attribution and MMM
Marketing attribution assigns credit to observed touchpoints. It helps explain which interactions preceded a conversion, but it cannot show how many purchases would have happened without the campaign. Marketing mix modeling estimates how channels and outside factors relate to aggregate outcomes. It can guide broader budget allocation and use experiment results for calibration.
| Method | Question it answers |
|---|---|
| Incrementality testing | Did this activity cause additional outcomes? |
| Attribution | Which observed touchpoints receive credit? |
| MMM | How do channels contribute across the full budget and time period? |
Start with the campaign where the wrong budget decision would cost the most. Choose the test design, calculate the required sample, and decide what result would change your next spend decision.