Synthetic training data for marketing models: real ads, model-written labels, and what that cost us
Synthetic training data can mean invented examples or model-written labels on real ones. What research and our own adapters show about each, with five checks.
By overads Inc., the team building gloofy
Synthetic training data is training material produced by a model rather than by people. For marketing work it comes in two kinds that behave very differently. One is invented examples: ads, briefs or conversations a model writes from scratch. The other is model-written labels on real material, such as a large model tagging real published ads. Our best results came from the second kind. The main risk of both is that the model you train inherits the generator's habits and its blind spots.
Two kinds of synthetic training data
The distinction matters more than the word "synthetic" suggests.
| Invented examples | Model-written labels on real inputs | |
|---|---|---|
| What is synthetic | The input and the answer | Only the answer |
| Typical use | Covering cases you lack, such as rare products, languages or formats | Labelling real material at a volume people cannot reach |
| Main risk | Inputs drift towards what models already write, narrowing what the trained model sees | The labeller's mistakes and preferences become your ground truth |
| How to check | Compare the spread of inputs with real ones | Audit a sample of labels against independent judges |
Most marketing teams reach for the first kind because it is easy: ask a model for a thousand ad variants and train on them. But real ads are cheap to collect and full of the variety a model will meet in use. Generating the input as well throws that variety away.
What research says about synthetic fine-tuning data
A study published in Findings of ACL 2026 by Max Schaffelder and Albert Gatt looked at what synthetic fine-tuning data does to a model, in three areas. Their abstract reports:
- Narrowing. Fine-tuning on synthetic data drawn from several different sources reduced distribution collapse, keeping the range of the model's outputs and the variety of its text wider than a single source.
- Safety. Both human-written and synthetic fine-tuning data could remove a model's safeguards. Synthetic data tended to give higher output quality, which the authors note makes the result more usable and also more dangerous.
- Self-preference. Fine-tuning reduced the model's bias towards its own outputs. Human data was most effective, followed by synthetic data from many sources.
The practical lesson is to use several generators, not one. A dataset written entirely by one model teaches the student that model's phrasing, structure and taste, and the student's outputs narrow accordingly.
What we learned from model-written labels
Every label in our training data is model-written. The ads are real published ads; the annotators who label them are Claude agents, not people, which we state on the site. So our models are a case study in the second kind of synthetic data, and the numbers are in our build log.
The labeller's consistency is the ceiling. Retraining our 4B model on labels from Claude Opus, an annotator that agreed with itself 96.7% of the time across three passes, lifted it from 0.796 to 0.808 on exam v3.3 of our benchmark. Moving to a 14B model on the same labels added only 0.008 more, reaching 0.816. Cleaner synthetic labels bought more than three and a half times the parameters. We covered this, and six other lessons, in our notes on LoRA fine-tuning.
Synthetic labels can be confidently wrong. When Claude Opus re-adjudicated our exam's original answer key, also written by Claude agents, it overrode 142 of 750 fields, about 19%. Agreement among model annotators is not the same as being right. Three agents can share a mistake.
Disagreement is how you find the errors. We found 84 inconsistent training labels by having three annotators judge the same ads blind and looking at where they split. Correcting them improved results more than doubling the data or moving to a model 2.4 times larger.
The catch: the labeller and the grader can share a source
There is a cost we have to state about our own results. Our two best models were trained on Claude Opus labels, and the current exam key was re-adjudicated by Claude Opus. On that exam, the student and the key share a teacher. That may flatter the student in the same way self-preference flatters a judge grading its own family's work, and it is a caveat on our top two rows.
It is also why the self-preference finding above matters for anyone distilling from one model. If one model writes your training labels and also judges your outputs, you cannot tell learning apart from agreement. Use a different model family, or people, for at least one of the two jobs. Our protocol for judges is in using an LLM as a judge.
Five checks before training on synthetic data
- Keep inputs real where you can. Real ads, real briefs, real queries. Generate only the parts you cannot collect, and label what is real.
- Measure the labeller's self-consistency. Run it three times on the same 100 items. If it disagrees with itself often, no model trained on it can be more consistent than it is.
- Audit with independent judges. Have two or three annotators, ideally from different model families or with a person, label a sample blind, and read every disagreement.
- Use several generators for invented examples. One model's style becomes your model's style. Mix sources, and check that outputs keep their variety.
- Separate teacher and grader. Never let the model that wrote your training labels also write your test key or judge your outputs without saying so.
Where gloofy fits, and where it does not
gloofy's released models are listed under the gloofy models. They are trained on real, public or authored examples; no customer or product data is used. Their labels are synthetic, written by Claude agents, and we publish that rather than describing them as human-labelled. The gloofy-14b adapter is the one trained on Claude Opus labels.
None of this makes gloofy a general model. The method above buys narrow, consistent structured work, such as tagging ad creative, and not broad reasoning. Whether fine-tuning is worth it at all for your task is a separate question, covered in when to fine-tune an LLM. More on training data and building models collects in the fine-tuning and building models hub.
What to do this week
- Sort your planned training data into real inputs and generated inputs, and replace generated inputs with real ones where you can collect them.
- Run your labelling model three times on 100 items and record its self-agreement.
- Label 50 items with a second model family or a person, and read every disagreement before you train.
- Write down which model wrote your labels and which wrote your test key. If they are the same, fix it or disclose it.
FAQ
What is synthetic training data?
Synthetic training data is material produced by a model rather than collected from people. It can be fully invented examples, such as generated ads or conversations, or model-written labels on real inputs, such as a large model tagging real published ads. The two carry different risks, so treat them separately when planning a fine-tune.
Is synthetic data good for fine-tuning an LLM?
It can be, with care. A Findings of ACL 2026 study found that synthetic data from several sources reduced the narrowing of a model's outputs compared with a single source. In our own work, cleaner model-written labels on real ads lifted a 4B model more than moving to a 14B model did.
What are the risks of synthetic training data?
The trained model inherits the generator's mistakes, style and preferences. One model's data can narrow outputs, model labellers can agree on wrong answers, and fine-tuning on any data can remove safeguards. If the same model writes your labels and your test key, your scores may reflect agreement rather than learning.
Should a large model label my training data?
It is a practical option, if you measure it. Check the labeller's self-consistency across repeated passes, audit a sample with an independent judge or a person, and correct disagreements before training. In our case, fixing 84 inconsistent labels improved results more than doubling the data or using a model 2.4 times larger.