Marketing AI benchmark: three public ones, what each measures, and how to pick

A marketing AI benchmark is a fixed, public test of how well a language model does marketing work, scored the same way for every model. There are only a few, and they measure very different things: analysing ad data with tools, producing creative ideas, and labelling real ads. Before a score tells you anything, check that the benchmark tests the job you are buying the model for, then check who wrote its answers.

Why most marketing AI scores cannot be checked

Most marketing AI products describe their quality with an evaluation nobody outside the company can see. That is not proof the numbers are wrong, but it means nobody can find out. A public benchmark publishes its items, its answer key or judging method, and its scoring code, so a claim can be rerun by someone with no stake in it.

General leaderboards do not fill the gap. They test reasoning, coding and knowledge, and a model that leads them can still mislabel what an ad is doing. Marketing has the same problem every specialised field had before it built its own tests: the general scores were never designed to answer its questions.

Three public marketing AI benchmarks compared

These are the three we know of with a published method. Each is a good test of something different.

Read the rows as answers to different questions. If you want an agent that pulls numbers from an ad account and explains them, AD-Bench is the closest test, and its results suggest the hardest multi-step tasks are still far from solved. If you want ideas, the Creativity Benchmark says the models are closer to each other than their marketing implies. If you want consistent labels on creative at volume, our public marketing benchmark is built for that and for nothing else.

Five checks before you trust a marketing AI benchmark

1. Does it test your task? A tagging score says nothing about strategy, and an analytics score says nothing about copy. Match the benchmark to the job, not to the vendor's headline.
2. Where do the items come from? Real requests and real published ads beat synthetic prompts, because synthetic items tend to resemble what models already produce. AD-Bench and marketing-bench both use real material.
3. Who wrote the answers? Answer keys have errors. Northcutt and colleagues found at least 3.3% label errors on average across ten widely used test sets, and at least 6% on ImageNet's validation set, enough to reverse which of two models looks better. A 2025 audit by Truong and colleagues sorts broken benchmark items into ambiguous questions, wrong keys and grading faults. Ours was worse: about 19% of our first key was wrong, which we covered in benchmark contamination and the other ways a score misleads.
4. If models are judged, who judges? The Creativity Benchmark compared three LLM-as-judge setups with its human ranking and found weak, inconsistent correlations, and its authors conclude automated judges cannot replace human evaluation for this work. Our own protocol for when an LLM judge is acceptable is in using an LLM as a judge.
5. Is the gap bigger than the noise? On 150 items, the top three results on our exam sit within about one standard error. Three training runs of one of our models, differing only by seed, spread 0.055 on the whole exam. A two-point lead on a small benchmark is often no lead at all.

What our own benchmark gets wrong

We should apply those checks to ourselves. marketing-bench tests one narrow job. It is keyed by Claude agents, not people, and the current key was re-adjudicated by Claude Opus. Our two best gloofy models were trained on labels from Claude Opus too, so on this exam they share a source with the key. That is the same self-agreement risk we flag for Claude's own entries, and it should be read as a caveat on our two best rows, not only on theirs.

Coverage is also incomplete. Gemini, GPT-5.6 and current Claude models were not measured on exam v3.3, so the current table says nothing about where gloofy sits against them. On the earlier exam, whose key we now know was about 19% wrong, Claude and Gemini scored above us. The corrections, and the claims we withdrew, stay visible in our build log.

Where gloofy fits, and where it does not

gloofy is a set of small open models built for structured marketing work such as tagging ad creative, and marketing-bench is the exam we built to measure that honestly. On that task, the released 14B adapter is tied with the 675B Mistral Large 3, as listed under the gloofy models. It has not been run on AD-Bench or the Creativity Benchmark, and we would expect it to do poorly on both: it is not an agent, and it is not built for open-ended ideas. Choosing a model for those jobs is covered in our guide to the best LLM for marketing.

The practical point is that no single benchmark ranks marketing models in general. More on measuring them properly collects in the benchmarks and evaluation hub.

What to do this week

• Write down the one task you most want a model for, and find the benchmark, if any, that tests it.
• Ask any vendor quoting a score for the items, the key and the scoring code. If none are public, treat the number as marketing.
• Build your own exam of 50 to 150 real examples, and have two people label 20 of them independently to see how often they agree.
• Treat differences inside your exam's noise as ties, and say so in any internal report.

FAQ

What is a marketing AI benchmark?

It is a fixed, public test that scores language models on marketing work in the same way for every model. Examples include AD-Bench for analytics agents, the Creativity Benchmark for ideas judged by practising creatives, and marketing-bench for tagging real ads. Each measures a different job, so no one benchmark ranks marketing models overall.

Which marketing AI benchmark should I use?

Use the one that tests your task. AD-Bench fits agents that query ad data with tools. The Creativity Benchmark fits idea generation, judged by human creatives. marketing-bench fits consistent labelling of real ad creative. If none matches, build a small exam from your own real examples and score candidate models on it.

Can benchmark answer keys be wrong?

Yes, often. A NeurIPS 2021 study found at least 3.3% label errors on average across ten common test sets, enough to change which model ranks higher. About 19% of the first answer key for our own ad-tagging exam was wrong, and correcting it moved the order near the top of our leaderboard.

Can an LLM judge marketing creativity instead of people?

Not reliably, according to the Creativity Benchmark. Its authors compared three LLM-as-judge setups with rankings from 678 practising creatives and found weak, inconsistent correlations and judge-specific biases. They conclude automated judges cannot substitute for human evaluation of marketing creativity, so use people for that, at least to check a sample.