Fine-tuning or RAG for documents in your language?
When retrieval (RAG) is the right answer, when training a model on your data is, and when you need both. With numbers from our public work.
We get this question on almost every first call. The short answer: retrieval (RAG) is for facts, fine-tuning is for behaviour. Often you need both. Below is the longer answer, and how to decide from a measurement rather than a hunch.
What is what
RAG (retrieval-augmented generation) finds the relevant passages in your documents before the model answers and adds them to the prompt. The model stays the same; what it sees changes.
Fine-tuning trains the model on your examples. The model itself changes: how it writes, which terms it uses, what format it returns and when it calls a tool.
When RAG is the right answer
- The content changes often. Price lists, policies, manuals, a knowledge base edited every week. You won't retrain a model for every change.
- You need a source. The answer has to cite the document and passage it came from.
- Access is restricted per document. Some users may only see part of the material. Retrieval can respect that; a model that memorised the content cannot.
- You need a fast start. A good retrieval pipeline takes days; training a model takes longer.
When fine-tuning is the right answer
- The output format is strict. Extraction into an exact JSON schema, classification into your categories, fields another system reads.
- Terminology and style. The model should write like your team, with your terms, not like a general assistant.
- Language. General models are weaker in smaller languages such as Slovenian: inflection, the dual, domain terms. They learn these from examples, not from a prompt.
- Tool calling. An agent must know which tool to call with which arguments, and also when to call none.
- A smaller model. On a narrow task, a tuned smaller model often matches a large one. It runs on your server, faster and without per-token fees.
When both
For agents, almost always. Retrieval brings fresh facts from the documents; the tuned model knows how to work with them: it reads the table correctly, answers in the right format and calls the right tool. Fine-tuning and retrieval are not rivals but two layers of the same system.
What our numbers show
Everything below is public, with model cards and results on Hugging Face.
| What we did | Measured |
|---|---|
| Continued pre-training on 1.78 billion Slovenian tokens (4B model) | perplexity −51 % |
| Fine-tuning Slovenian speech recognition | word error rate 46.8 % → 22.3 % |
| Tool-calling fine-tune, 2,471 held-out steps | correct "no call" decisions 0.847 → 0.925 |
The last row matters most for agents. After tuning, the model knows better when it must not call a tool. Retrieval can't give you that, because it is a question of behaviour, not facts.
Cost and speed
- RAG needs no training, but every call is longer: the prompt carries the retrieved passages, so more tokens and more waiting. With a third-party API you pay for that on every call.
- Fine-tuning costs once up front (and at each retrain), then every call is shorter and cheaper, especially with a smaller model on your own server.
Which one pays off depends on volume. At a few hundred calls a month an API is often cheaper. At thousands a day the sums flip.
How to decide
Not by opinion, by measurement:
- Collect fifty real examples of the task with correct answers.
- Freeze them as an evaluation set nobody changes.
- Measure the untuned model, then the model with retrieval, then a tuned model, all on the same set.
- Pick the simplest option that reaches the level you need.
Sometimes the result is that training isn't needed and good retrieval is enough. That is a good result too, because you learned it for the price of one day, not a project.