When to fine-tune an LLM: our untrained base model won four of five tasks
When to fine-tune an LLM, measured on five marketing tasks where the untrained base beat our fine-tune four times, and the rule that explains why.
By overads Inc., the team building gloofy
When to fine-tune an LLM comes down to one question: can you write the knowledge down? If you can state it in a sentence (a character limit, a currency rule, where a phone's interface covers a video), put it in the system prompt. Fine-tune only when the skill is tacit, something people recognise but cannot specify. We learned this the expensive way: across five marketing tasks we fine-tuned a 4-billion-parameter model for, the untrained base model beat our fine-tune in four of them.
What we measured, task by task
We trained one LoRA adapter per task on Qwen3-4B (why one per task, and six other lessons, are in our notes on LoRA fine-tuning), then ran each adapter and the untouched base model through the same frozen exams. Prose outputs were judged blind, by three judges, with every pair shown in both orders so position could not decide the winner (the reasons are in our post on using an LLM as a judge). Here is the result for each of the five task families.
| Task | Untrained base | Fine-tune | Winner |
|---|---|---|---|
| Captions: staying within the platform length | 0.89 | 0.83 | Base |
| Campaign setup: using the local currency | 0.76 | 0.34 | Base |
| Campaign setup: getting the arithmetic right | 0.09 | 0.02 | Base |
| Image direction: respecting Meta's 9:16 safe zones | 9 of 10 | 5 of 10 | Base |
| Campaign diagnosis, blind pairwise judging | 9 wins | 1 win | Base |
| Video hooks, blind pairwise judging | 9 wins | 36 wins | Fine-tune |
The single win is real. On writing the opening three seconds of a short-form video ad, the fine-tune won 36 to 9 on a 49-item exam, with order consistency of 49 out of 49 and a one-sided binomial p of 0.000033. It is the only pairwise judging result in our build log that clears statistical significance. Everything else in the table went the other way.
Why the fine-tune lost: facts belong in the prompt
Look at what the four losses have in common. Safe zones, local currency, length limits and the arithmetic of a budget split are all facts. A capable base model that is told a fact in its instructions applies it. Training the same fact into the weights did not merely fail to help; it made the model worse at following instructions.
The clearest symptom was in campaign setup. The fine-tune fell into verbatim repetition in 17 of 88 answers, a failure the base model never showed once. That is what losing instruction-following looks like in practice: the model stops responding to the brief in front of it and starts replaying patterns from training.
Hooks are different. Nobody can write down the rule for an opening line that stops a scroll. Good hooks share a feel that editors recognise and cannot specify, which is exactly the kind of knowledge that compresses well into weights from examples. That is where fine-tuning earned its cost.
So the rule we now work by is short:
- Fine-tune for tacit, compression-shaped skill: taste, tone, recognising a pattern from many examples.
- Prompt for anything you can state: specs, limits, policies, formats, the current date, the client's currency.
When to fine-tune an LLM: a five-question check
Before training anything, we now ask these in order. Stop at the first "no".
- Have you tried stating it? Write the rule into the system prompt with two or three examples, and measure that version first. It is the baseline the fine-tune has to beat, not a step to skip.
- Is the knowledge tacit? If an expert could write it as a checklist, it is a fact, and the prompt is the right home for it.
- Is the task narrow and repeated? Fine-tuning pays when the same job runs thousands of times. A model handling many different jobs is better served by good instructions.
- Do you have a frozen exam the training data never touched? Without one, you cannot tell a gain from leakage. Our own public marketing benchmark has zero training overlap for this reason.
- Will the gain survive the noise? See the next section. If the difference you expect is smaller than the spread between training runs, you will not be able to see it.
The part most fine-tuning guides skip: noise
Three training runs that differed only by random seed spread 0.055 on whole-exam scores, and up to 0.34 on individual ten-item slices. That is large enough to manufacture a win or a loss on any small subset, and it is why we formally retracted several per-slice claims from earlier versions of our own results.
It also cuts the other way: a gap well clear of the noise is real, and one inside it is not. On exam v3.3 of our benchmark, 150 real published ads, our released 14B adapter scores 0.816 against 0.825 for Mistral Large 3. At 150 items the top three results sit within about one standard error of each other, so we describe them as tied, not ranked, even though the table puts ours second.
The practical lesson: run any fine-tune at least three times with different seeds before deciding it helped, and measure the spread for your own setup. A noise figure measured on someone else's data does not transfer.
Fewer places to fine-tune a closed model
The decision is getting made for some teams anyway. OpenAI's deprecations page says that from 7 May 2026 organisations that had never fine-tuned could no longer start, that inactive organisations lost access on 2 July, and that from 6 January 2027 no customer can create new fine-tuning jobs, while existing fine-tuned models keep serving until their base model is retired. According to Tessl, OpenAI told developers its newer models handle many tasks that used to need fine-tuning, and that prompting is now cheaper and faster for most of them.
Our numbers agree with the direction of that, for facts. They disagree for tacit skill: the hooks result is a case where training beat prompting by a wide, significant margin on a 4B model. If you have a job like that, open-weights models are where LoRA fine-tuning remains fully in your hands, and where you can keep the adapter, the exam and the result.
Where gloofy fits, and where it does not
We applied the rule to our own product. Four of the five task families now ship as the base model plus a task-specific system prompt, with no adapter at all, and campaign diagnosis ships as the unmodified base because it beat our adapter 9 to 1. The fine-tuning budget goes to the jobs where knowledge is tacit.
gloofy-1-nano, the released 4B model, is small enough to run on a laptop under Apache 2.0 (what that buys you is covered in choosing a local LLM for marketing), and that is its point: cheap, private, consistent structured work. It is not a match for frontier models on general reasoning. On the last version of our exam that included them, Claude and Gemini scored above it on ad tagging and GPT-5.5 tied it. A larger 8B tier scored 0.741 against the 4B model's 0.780 on identical data, so we shelved it rather than ship a bigger number of parameters with nothing to show for them.
The same logic applies to platform tools. When Meta's assistant asks for your brand guidelines, as covered in our post on what the new Meta AI ad tools actually ship, it is betting on facts in context, not facts in weights. That is the right bet. More results like these, including the ones that went against us, collect in the fine-tuning and building models hub.
What to do this week
- List the rules your marketing AI keeps breaking. If each one fits in a sentence, move it into the system prompt before anyone suggests training.
- Build a frozen exam of 50 to 150 real examples that no prompt or training set has seen, and score the prompted base model on it first.
- Reserve fine-tuning for the one or two jobs where your best people say "I know it when I see it".
- If you do fine-tune, train three seeds and report the spread alongside the score.
FAQ
When should you fine-tune an LLM instead of prompting it?
Fine-tune when the skill is tacit and repeated: something experts recognise from examples but cannot write as rules, such as what makes a video hook work. If the knowledge can be stated as a fact, like a character limit or a currency rule, put it in the system prompt. In our tests the prompted base model won those tasks.
Can fine-tuning make a model worse?
Yes. Training facts into a model can cost it instruction-following. Our fine-tune for campaign setup dropped from 0.76 to 0.34 on using the right local currency and fell into verbatim repetition in 17 of 88 answers, something the untrained base model never did. Always compare against the prompted base model on the same exam.
How many training runs do you need to trust a fine-tuning result?
At least three with different random seeds. In our runs, seed alone moved whole-exam scores by 0.055 and ten-item slices by up to 0.34. A gain smaller than that spread cannot be told apart from chance, so measure the spread for your own data before claiming an improvement.
Is fine-tuning still available for closed models?
Less than before. OpenAI's deprecations page says no customer can create new self-serve fine-tuning jobs from 6 January 2027, after earlier restrictions for new and inactive organisations. Open-weights models remain fully trainable with methods such as LoRA, and you keep the resulting adapter.