Creative testing is a sampling problem

Here is a programme that looks disciplined and is not. Every fortnight the team ships four new ads, watches them for a week, keeps the best one and kills the other three. Everyone agrees this is rigorous. After a year the account has run a hundred ads and the team cannot tell you a single thing they have learned about what works.

The unit of learning is the batch, not the ad

The problem is that individual ad performance is mostly noise. Even at reasonable spend, the difference between the best and worst of four ads is usually within the range you would get by running the same ad four times. Picking the winner is, much of the time, picking the luckiest sample.

What is not noise is the difference between kinds of ad. Testimonial versus demonstration. Price-led versus problem-led. Presenter to camera versus product only. Those differences are large enough to detect, and they are the only findings that transfer to the next quarter.

Individual ad performance is mostly noise. The difference between kinds of ad is not.

How to structure it

• Define three or four concepts — a claim about what will persuade this audience.
• Produce three to five executions of each concept, deliberately varied in surface detail.
• Judge at the concept level by pooling the executions. Never promote a single execution on its own result.
• Retire a concept only when it loses across two independent batches.

What this costs you

More production, and a slower read. You are shipping twelve to twenty assets per cycle rather than four, and you are waiting two cycles before concluding anything. In exchange you get findings that hold, a creative brief that improves quarter on quarter, and an argument you can actually win with a creative director.

The failure mode to watch

Executions within a concept drifting until they are no longer the same concept. If your “problem-led” batch contains one ad that opens on price, you have contaminated the pool. Write the concept definition down before production and check the assets against it before launch, which sounds bureaucratic and takes about ten minutes.