Creative testing: how much data a winning ad needs, and why five identical ads still produce one
Creative testing needs more data than most tests get. Sample sizes for CTR lifts, why five identical ads still produce a winner, and how Meta tests work.
By overads Inc., the team building gloofy
Creative testing means showing several versions of an ad to comparable audiences and keeping the one that performs best. Each version needs enough data for chance to be ruled out. That is usually far more than tests get: spotting a 20% lift in a 1% click-through rate takes about 43,000 impressions per version. With less, a set of five ads will almost always produce a "winner", even when all five are identical.
Why creative testing matters more as platforms make more ads
Platforms now generate variants faster than anyone can judge them. Meta's Advantage+ creative and Ads Creative Studio produce versions on request, and TikTok's Smart+ can pick from creator videos, your own posts and AI-generated variants in one pool. More versions means a smaller budget per version, and less data per version means a winner chosen more by luck.
That is not an argument against generated creative. It is an argument for knowing how much evidence a test produces before you act on the result.
How Meta's creative testing tool works
According to Jon Loomer's walkthrough, last updated in January, Meta's tool sits at the ad level in Ads Manager and runs like this:
- You create two to five test ads, as duplicates you edit. Ads that are already live cannot be tested directly.
- A share of the budget is set aside for the test, and Meta recommends no more than 20%.
- During the test, Meta aims to spend a similar amount on each ad. The default length appears to be seven days.
- The campaign must use the Highest Volume bid strategy, not a cost per result goal, bid cap or ROAS goal.
- You compare ads on cost per result, CPC, CPM or a cost per conversion event, in the Experiments section.
- When the test ends, the ads keep running, but Meta stops holding spend level, so it will not necessarily give the most to your best performer.
Even spend is the useful part. Without it, Meta's delivery system decides which ad gets budget early, and an ad starved of impressions in its first days never gets a fair reading.
TikTok works the other way. Its help page on auto-selected creatives, last updated in April, says Smart+ ranks creatives by past and predicted performance and changes delivery in real time. You can review and reject videos, set exclusion rules, or switch to custom selection to opt out. It is currently available for app promotion, and for lead generation with select customers only. That is optimisation, not a controlled test: the system spends more on what it predicts will win, so the results show its preferences as much as your audience's.
How many impressions does a creative test need?
These are standard two-proportion sample sizes, at 95% confidence and 80% power, for the impressions each version needs to detect a relative lift in click-through rate.
| Baseline CTR | Lift you want to detect | Impressions per version |
|---|---|---|
| 1% | 10% (1.0% to 1.1%) | about 163,000 |
| 1% | 20% (1.0% to 1.2%) | about 43,000 |
| 1% | 30% (1.0% to 1.3%) | about 20,000 |
| 2% | 10% | about 81,000 |
| 2% | 20% | about 21,000 |
| 2% | 30% | about 10,000 |
Two things follow. Small lifts are expensive: halving the lift you want to see roughly quadruples the data needed. And if you judge on conversions rather than clicks, the counts are far smaller, so the same test needs much longer. A version with 40 conversions against another with 32 is not a result.
Five identical ads will still produce a winner
To show how often chance alone produces a winner, we ran a simple simulation. Five ads, all with the same true click-through rate of 1%, each shown 10,000 times, repeated 2,000 times with a fixed random seed. In 71% of runs, the best ad beat the worst by 20% or more, and the median gap between best and worst was 26%. None of the ads was better.
This is the multiple comparison problem, and it gets worse with more variants. Picking the best of five is not the same as comparing two: the more versions you test, the larger the gap chance will produce between the top and bottom. When a tool lets you test five versions, the honest move is to compare your best against your control, with a sample size planned for that comparison, rather than ranking all five by their raw results.
We learned the same lesson on our own model work. Three training runs of one model that differed only by random seed spread up to 0.34 on ten-item slices of our exam, enough to invent a win on any small subset, and we withdrew several per-slice claims because of it. The details are in our build log. Ads are no different: a small slice looks decisive and usually is not.
Large effects can still show up quickly. Our one significant result in blind judging, on video ad hooks, came from only 49 items because the gap was wide: 36 wins to 9, with a one-sided binomial p of 0.000033. The point is not that small tests are useless, but that they can only detect big differences.
A creative testing plan that survives the statistics
- Pick the metric before the test. One primary metric, chosen in advance. Changing it after the results arrive is how chance becomes a finding.
- Size the test. Use the table above or any two-proportion calculator. If your budget cannot reach the impressions needed, test fewer versions or look for a bigger difference.
- Change one thing. A test that swaps the hook, the offer and the visual at once tells you that a set won, not why. Our guide to Performance Max asset testing covers how to tag what changed.
- Run the full window. Do not stop when one ad pulls ahead on day two.
- Retest winners. A real winner wins again against a fresh control. A lucky one regresses.
- Keep a log. Write the result as a rule about what changed, and count how often each rule holds across tests.
Where gloofy fits, and where it does not
gloofy does not run tests, connect to Meta or TikTok, or predict which ad will win. No model should be trusted to replace the test itself, and ours has never been measured on predicting ad performance.
What it does is the labelling step that makes tests add up: the same hook, angle, persona, offer and funnel-stage tags on every variant, run on your own machine. That is the job described under what gloofy does. Tagging consistently means a quarter's worth of small tests can be pooled by what changed, which helps with the sample size problem above. More on hooks, angles and testing collects in the ad creative and copy hub.
What to do this week
- Look at your last three creative tests and count the impressions or conversions per version. Compare them with the table.
- If you use Meta's tool, test two or three versions instead of five unless your budget covers the larger sample.
- Treat TikTok's auto-selection as delivery optimisation, and run a custom-selection test when you need an answer.
- Rerun your last "winner" against a fresh control before you build next quarter around it.
FAQ
What is creative testing in advertising?
Creative testing means running two or more versions of an ad to comparable audiences, with spend held similar, to learn which performs better on a metric chosen in advance. A useful test changes one element at a time, runs long enough to gather adequate data, and records the result as a rule you can test again.
How many impressions do you need for an ad creative test?
It depends on your baseline and the lift you want to detect. At a 1% click-through rate, detecting a 20% relative lift needs about 43,000 impressions per version, at 95% confidence and 80% power. A 10% lift needs about 163,000. Conversion-based tests need far more traffic, because conversions are much rarer than clicks.
How does Meta's creative testing tool work?
According to Jon Loomer's guide, you create two to five test ads in Ads Manager, set aside part of the budget (Meta recommends no more than 20%), and Meta spends similarly on each, by default for seven days. It requires the Highest Volume bid strategy. After the test, ads keep running without even spend.
Why does my winning ad stop working?
Often it was never better. When several similar ads run on small samples, the best one usually wins partly by chance, and that luck does not repeat, so performance falls back towards the average. In our simulation of five identical ads, the best beat the worst by 20% or more in 71% of runs. Retest winners against a fresh control.