Most teams fine-tune a model when what they needed was to hand it a document

A company decides it wants an assistant that knows its own products, policies and history. The first instinct, almost without exception, is to train a model on the company's documents. It sounds obviously correct — teach the machine what we know — and it is usually the most expensive way to get a worse result than a much simpler approach would have produced in an afternoon.

The reason is a distinction that rarely gets drawn clearly. There are three ways to make a general model do a specific job, and they repair different faults. Choosing between them is the single highest-leverage decision in an applied AI project, and it is made wrongly far more often than not.

Three levers, not one

Prompting puts instructions and material into the model's input. Nothing about the model changes; you are simply telling it what to do and giving it what to work with. It is instant, free to iterate, and reversible. Its limits are the size of the context window and the fact that every token you send is paid for on every single call.

Retrieval is prompting with a search step in front of it. When a question arrives, the system finds the relevant passages from your documents and places them in the context, so the model answers from text that is actually present rather than from memory. The model still has not changed.

Fine-tuning actually modifies the model's weights by training it further on examples of the task. This is the only one of the three that changes the model itself, which is why it is also the only one that is slow, costly and awkward to undo.

What fine-tuning is genuinely good at

Fine-tuning teaches a model how to respond. That covers more than it sounds like.

It is the right tool for form: a house style, a consistent tone, a strict output schema that must parse every time, a particular way of structuring a clinical note or a legal summary. It is right for task shape — an unusual instruction pattern the model keeps drifting away from, or a classification scheme specific to your organisation. It is right for domain register: getting a model to write the way people in a specialist field actually write, rather than the way a general assistant imitates them.

And it is the right tool for cost. If a large model does your task well and a small one does not, fine-tuning the small model on the large one's outputs — distillation — can give you most of the quality at a fraction of the price and latency. For a service handling steady, repetitive traffic, this is frequently where the real money is.

Why fine-tuning facts goes wrong

Facts are the case where the instinct fails, and it fails for two separate reasons.

The obvious one is maintenance. Weights do not update themselves. A fact trained into a model is frozen there: when the price changes, the policy is revised or the product is discontinued, you have a model confidently asserting last quarter's answer with no way to correct it short of training again. Worse, it cannot cite anything. A retrieval system can show the passage it used; a fine-tuned model can only tell you what it now believes.

The less obvious reason is more damaging, and it surprises people. Research on this has found that when you fine-tune a model on facts it did not already know, those examples are absorbed slowly — and as they are absorbed, the model's tendency to hallucinate on other questions rises. The explanation is uncomfortable but sensible: the model cannot learn "this is true" as an isolated fact. What it learns from the training signal is a behaviour — produce a confident, specific assertion of this shape — and it then applies that behaviour to neighbouring questions where it has no knowledge at all. You set out to teach it something and instead taught it to sound certain.

Retrieval changes what a model can see. Fine-tuning changes how a model behaves. Most failed projects asked one of them to fix a problem belonging to the other.

How fine-tuning is actually done

Few people now retrain every weight. Parameter-efficient methods dominate, and the most common freezes the original model entirely and trains a small pair of low-rank matrices alongside each layer — an adapter that captures the adjustment without touching the base. The result is a file measured in megabytes rather than gigabytes, several of which can be kept for different tasks and swapped at load time, and which can be run on modest hardware.

Two practical facts govern whether it works.

Data quality dominates data quantity. A few hundred carefully constructed examples that demonstrate exactly the behaviour you want will routinely outperform tens of thousands of scraped, inconsistent ones. Fine-tuning amplifies whatever pattern is in the data, including the sloppiness. Most of the work in a successful fine-tune is curation, not training.

Narrow training degrades general ability. Push a model hard towards one task and it gets worse at others — catastrophic forgetting. A model fine-tuned into an excellent extractor of one document type may become noticeably worse at ordinary conversation, reasoning or another language. If your system needs both, that trade has to be measured rather than assumed.

The order to try things in

The practical sequence is almost always the same, and it is cheapest first.

Start with the prompt. A surprising share of problems that get diagnosed as "the model isn't good enough" are instruction problems, and they disappear when the task is specified properly with a few worked examples.

If the answer depends on information the model cannot be expected to hold — your documents, current data, anything that must be cited or that changes — use retrieval. This is not a lesser option; for factual grounding it is the correct one, and no amount of fine-tuning substitutes for it.

If the model can clearly do the task but keeps doing it in the wrong shape, and prompting has not fixed it after honest effort, fine-tune for behaviour. If it does the task well but too slowly or expensively, fine-tune a smaller model on the larger one's output.

If it lacks a genuine capability — reasoning in a specialist domain, competence in a language it barely saw — that is the hard case, and it usually needs more than a task fine-tune. This is worth saying plainly because it is the situation most relevant to Indian-language work, where the honest answer is often that adaptation cannot rescue a base model that was never trained on enough of the language.

Underneath all of it sits a requirement that is skipped constantly: build the evaluation set before you change anything. Without a fixed set of realistic cases scored the same way each time, you cannot tell whether a fine-tune helped, and teams routinely ship adaptations that made things worse in ways nobody measured.

Why it matters for students and researchers

Very little frontier pretraining will ever happen in most organisations, in India or anywhere else. Almost all applied work is adaptation — taking a capable general model and making it do one job reliably, cheaply and within a latency budget. That is the job description, and the scarce skill in it is not the ability to run a fine-tuning script, which is now a handful of lines.

The scarce skill is diagnosis: looking at a failing system and correctly identifying whether it is short of information, short of instruction, short of capability, or simply badly evaluated. Those four have different fixes and similar symptoms. Getting that judgement right saves months, and getting it wrong produces the most common artefact in the field — an expensively fine-tuned model that confidently gives outdated answers and cannot tell you where any of them came from.

Frequently asked questions

What is the difference between fine-tuning and RAG?

Fine-tuning trains the model's weights on examples, changing how it behaves. Retrieval-augmented generation leaves the model unchanged and instead searches your documents at query time, placing relevant text into the input so the model answers from material that is present.

Should I fine-tune a model on my company's documents?

Usually not. Facts trained into weights become stale, cannot be cited and cannot be updated without retraining. Retrieval is the correct tool for knowledge; fine-tuning is for behaviour, format and cost reduction.

Does fine-tuning on new facts cause hallucination?

It can. Studies have found that examples teaching genuinely new knowledge are learned slowly, and that as they are learned the model becomes more likely to fabricate answers to unrelated questions — because what it absorbs is the behaviour of asserting confidently rather than the fact itself.

What is LoRA?

It is a parameter-efficient fine-tuning method that freezes the original model and trains small low-rank adapter matrices alongside it. The resulting files are small, can be swapped between tasks, and can be trained on far less hardware than full fine-tuning requires.

How much data does fine-tuning need?

Less than most people expect, if it is good. A few hundred carefully written examples demonstrating the exact desired behaviour typically beat tens of thousands of inconsistent ones, because fine-tuning reproduces whatever patterns exist in the training data, including its flaws.