What is LLM fine-tuning, and when don't you need it?

What is fine-tuning, and what does it change?

Fine-tuning an LLM means training an existing model further on your own examples, which changes the model itself. A large language model is, underneath, billions of numeric weights learned during its original training. Fine-tuning runs a second, much smaller round of training on your examples, nudging those weights so that the model’s default behaviour shifts towards what the examples show.

Compare that with the other ways of customising a model:

Method	What changes	Effort	Good for
Prompting	The instructions in each request	Hours	Most tasks; always the first step
Few-shot examples	Worked examples added to each prompt	Hours	Formats and edge cases
RAG	Relevant documents added to each prompt	Weeks	Company knowledge, current facts, citations
Fine-tuning	The model’s weights	Weeks, plus upkeep	Consistent behaviour, narrow tasks, smaller models
Pre-training from scratch	Everything	Months and millions of dollars	Almost never a business decision

The famous demonstration of what fine-tuning can do is OpenAI’s InstructGPT work (Ouyang et al., 2022). A 1.3 billion parameter model fine-tuned on human demonstrations and preferences produced outputs people preferred over the original 175 billion parameter GPT-3, despite having 100 times fewer parameters. The lesson for businesses: fine-tuning’s biggest payoff is often making a smaller, cheaper model behave well on a specific job.

How does fine-tuning work, step by step?

You collect examples of the input and the ideal output, train on them, then test the new model against a held-back set. In practice:

1. Define the task narrowly. “Extract these 14 fields from supplier invoices as JSON”, not “be better at finance”.
2. Build the dataset. Hundreds to thousands of example pairs, reviewed by someone who knows what a correct answer looks like. Remove personal information you don’t need.
3. Hold back a test set the model never trains on, plus a baseline score from the best prompt-only approach.
4. Choose a method (see the next section) and a base model.
5. Train. On a managed platform this is a job you submit; on your own infrastructure it’s a GPU run.
6. Evaluate against the held-back set and the baseline. If it doesn’t clearly beat good prompting, stop.
7. Deploy and monitor, and keep the dataset, because you’ll retrain when the base model is retired.

What are the main types of fine-tuning?

The methods differ in what kind of feedback the model learns from and how much of it gets retrained.

Method	What you provide	Typical use
Supervised fine-tuning (SFT)	Prompt and ideal response pairs	Formats, extraction, classification, house style
Preference tuning (for example DPO)	A better and a worse response for each prompt	Tone, helpfulness, avoiding unwanted styles
Reinforcement fine-tuning	Prompts plus a grader or reward function	Multi-step reasoning tasks with checkable answers
Distillation	Outputs from a large “teacher” model	Making a small model match a big one on your task
LoRA and other parameter-efficient methods	Any of the above, training small add-on weights	Cheaper training and hosting of many variants

Parameter-efficient methods changed the economics. LoRA (Hu et al., 2021) freezes the original weights and trains small add-on matrices instead; against full fine-tuning of GPT-3 175B, it cut trainable parameters by 10,000 times and GPU memory by 3 times. QLoRA (Dettmers et al., 2023) added 4-bit quantisation and showed a 65 billion parameter model could be fine-tuned on a single 48 GB GPU. Managed services have followed: Amazon Bedrock, for example, offers supervised fine-tuning, reinforcement fine-tuning and distillation for selected models. OpenAI’s platform added supervised, preference (DPO) and reinforcement fine-tuning, but at the time of writing OpenAI says it is winding down its fine-tuning platform and no longer accepts new users.

Fine-tuning examples: what is it used for?

Fine-tuning is used where the task is narrow, repeated at volume and easy to show with examples. Typical business examples:

• Classification and routing: labelling support tickets, emails or claims into a fixed set of categories.
• Structured extraction: pulling the same fields from invoices, forms or contracts into JSON every time.
• House style and tone: drafting replies or summaries in a consistent voice that’s hard to describe in a prompt.
• Domain language: handling specialist terms in areas like medicine, law or engineering more reliably.
• Smaller, cheaper models: distilling a large model’s behaviour on one task into a small model that’s faster and cheaper to run, or that can run on your own infrastructure.

What these have in common is that they change how the model responds. None of them is “make the model know our documents”, which is a retrieval job.

Fine-tuning vs RAG: the short version

Fine-tuning changes behaviour; RAG supplies knowledge. They solve different problems and are often combined.

Question	Fine-tuning	RAG
What changes?	The model’s weights	The information in each prompt
Best for	Format, tone, narrow tasks	Company knowledge, current facts
Can it cite sources?	No	Yes
Keeping it current	Retrain	Update the index

When does a business need fine-tuning?

When you have a stable, well-defined task, a measured gap that prompting and retrieval can’t close, and enough good examples. Signals that it may be worth it:

• A very high volume of one task, where a fine-tuned small model would cut inference cost or latency substantially.
• Output must follow a strict structure or classification scheme every time, and prompting still fails on too many cases.
• A specialised style or domain language that’s hard to describe but easy to show.
• You need to run on a small open-weight model inside your own environment for sovereignty reasons, and it needs help to match a larger hosted model. See open-weight vs API models.

And when you don’t:

• You want the AI to know your documents. Ovadia et al. (2023) found RAG consistently outperformed unsupervised fine-tuning for adding knowledge, both familiar and new. Use retrieval.
• Your facts change monthly or weekly. Every change would need retraining.
• You need citations. A fine-tuned model can’t point to where it learned something.
• You haven’t tried a strong prompt with examples. Many “we need fine-tuning” problems disappear here.
• You don’t have an evaluation set. Without one you can’t tell whether the fine-tune helped.

For a full side-by-side decision, read RAG vs fine-tuning.

What goes wrong with fine-tuning?

The common failures are data quality, weak evaluation and underestimating upkeep.

• Garbage in, confident garbage out. Inconsistent examples teach inconsistent behaviour, delivered fluently.
• Forgetting general skills. Heavy training on a narrow task can degrade other abilities. Test beyond the target task.
• Baked-in facts go stale. Anything factual the model learned is frozen at training time, and may be stated confidently after it changes, which feeds hallucinations.
• Base model lock-in. When the vendor retires the base version, the fine-tune goes with it.
• Privacy in the weights. Personal information in training data can be memorised and reproduced. Minimise it and keep the model’s access as controlled as the data’s.
• Hidden running costs. Custom models may need dedicated capacity, and on Bedrock, for example, you pay for training tokens multiplied by epochs plus monthly storage per custom model. See LLM running costs.

Worked example: should this team fine-tune?

A claims team wants a model to read incident emails and output a claim category, urgency and five extracted fields.

1. Baseline: a well-written prompt with 10 examples on a mid-sized hosted model scores 91% on 300 held-back emails.
2. Target: 97%, and volume is 40,000 emails a month.
3. Option A: move to a larger model. Accuracy rises to 95%, but cost per email roughly triples.
4. Option B: fine-tune a small model on 2,000 reviewed examples. If it reaches 97% at a fraction of the per-email cost, the training and upkeep pay for themselves.

The figures here are illustrative, but the logic is the point: fine-tuning was justified by a measured gap, a stable task and high volume, not by a feeling that the model should “know us better”.

How All Webbed Labs approaches fine-tuning

We treat fine-tuning as a late optimisation, not a starting point. We build the evaluation set first, push prompting and retrieval as far as they go, and fine-tune only when the numbers show a gap worth the upkeep. Where a fine-tuned open-weight model needs to run inside your own cloud account in an Australian region, that’s part of our private LLM deployment work, measured using the approach in our guide to LLM evaluation.