How do you measure incrementality for ecommerce ads?

Everything else is description. A report showing a high return on a campaign is telling you that people who saw the advertising bought; it cannot tell you whether they would have bought regardless, and for retargeting and brand campaigns the answer is often that many of them would.

So the design always has the same shape: two comparable groups, one exposed and one not, and a comparison of total outcomes rather than attributed ones. What varies is how you form the groups.

The outcome measured has to be total store revenue or total orders from each group, not campaign conversions. If you measure the campaign, the withheld group has no campaign and therefore no data, and the test measures nothing.

That point is worth labouring because it is the most common way these tests fail. Somebody pauses a campaign, sees that its conversions went to zero, and concludes the campaign was working.

A matched regional split is usually the best first test. Divide your market into regions, pair them by past revenue and traffic so each pair is similar, then turn the advertising off in one of each pair. Compare total revenue between the two sets over the test period.

A budget step test is easier and less rigorous. Raise or lower spend on one channel by a meaningful amount, hold everything else constant, and compare total store revenue before and after against the same period in a comparable earlier window. It cannot separate seasonality perfectly, and it answers the question you usually care about: what did the last increment buy.

Platform lift studies are the third route. Some advertising platforms will run a controlled experiment for you, withholding advertising from a randomly selected group and reporting the difference. These are well designed and limited to that platform's own inventory.

An audience holdout is the fourth: exclude a random share of your customer list from a campaign and compare their purchasing. This suits retargeting and email particularly well, and it is the cheapest test to run.

Most incrementality tests fail because they could never have detected anything. The effect being measured is usually a modest percentage of total revenue, and total revenue is noisy, so a small test over a short period produces a result indistinguishable from normal variation.

Work out in advance what size of difference you would need to see, and whether the volume in the test period could show it. If a channel represents a small share of your revenue, a test on it needs a long run or a large geographic split to be readable.

Run for at least a full purchase cycle plus a couple of weeks, and cover whole weeks so the weekday pattern balances. Avoid periods containing a sale, a launch, a stock-out, or a major change in another channel.

And control the contamination. If a paused channel's demand simply flows to another campaign that picks up the same queries, the test measures reallocation rather than incrementality. Set exclusions in the other campaigns before you begin.

Write the decision rule before the test starts: the size of revenue difference that would make you increase the budget, and the size that would make you cut it. Deciding afterwards is how a test becomes a search for support.

Expect the result to be less flattering than the platform report and more flattering than the sceptical case. Most channels produce some incremental revenue and less than they claim, and the useful output is a ratio you can apply to future reporting rather than a verdict on the channel.

Use that ratio as a discount factor. If a test suggests a channel's reported revenue overstates its contribution by a known amount, you can keep using platform reporting for day-to-day decisions and apply the discount when comparing channels for budget.

Then repeat it periodically. Incrementality changes with the market, the season, the mix of brand and non-brand demand, and how saturated the channel is. A result from eighteen months ago describes a business that no longer exists.