RAG vs fine-tuning: what each does, and which one your AI project needs
RAG vs fine-tuning is a choice between giving a language model your knowledge at question time and changing the model itself. Retrieval-augmented generation (RAG) finds relevant passages in your documents and hands them to the model with each question. Fine-tuning trains the model further on your examples so it behaves differently. This guide explains how each works, compares them side by side, summarises what research has found, covers the data protection side and gives a short test for choosing, including when to use both.
The short version
- RAG adds knowledge; fine-tuning shapes behaviour.
- For questions answered from company documents, start with RAG: it updates instantly, cites sources and respects access rights.
- Use fine-tuning for a consistent format, tone or narrow task that good prompts cannot reach.
- Try prompt engineering first; it is the cheapest of the three.
- Many production systems combine a tuned model with RAG.
Definitions
Retrieval-augmented generation (RAG) combines a language model with a search step. Your documents are split into passages and indexed, usually as vectors that capture meaning. When a question comes in, the system retrieves the most relevant passages, adds them to the request and asks the model to answer from them. The model itself is unchanged. The term comes from a 2020 research paper by Lewis and colleagues, which combined a pre-trained model's "parametric memory" with a "non-parametric memory", a searchable index, and noted that providing provenance and updating a model's knowledge were open problems that the index could address.
Fine-tuning continues training a pre-trained model on a smaller set of your own examples, such as questions with ideal answers, documents in a house format or labelled cases. The model's weights change, so the new behaviour is built in and needs no examples in each request.
Prompt engineering changes neither: it improves the instructions and examples sent with each request.
RAG vs fine-tuning side by side
| Aspect | RAG | Fine-tuning |
|---|---|---|
| What changes | The context given to the model | The model's weights |
| Best for | Facts, documents, policies, product data | Format, tone, terminology, narrow tasks |
| Updating knowledge | Re-index the changed documents | Retrain and re-test |
| Citing sources | Yes, the retrieved passages | No, knowledge is mixed into the weights |
| Access control | Per document and user, at retrieval | None inside the model |
| Deleting personal data | Remove from the index | Hard; may require retraining |
| Data needed | Your existing documents | Hundreds or more curated examples |
| Main failure | Wrong or missing passages retrieved | Outdated or invented facts stated fluently |
What research has found
A 2023 study by Ovadia and colleagues compared unsupervised fine-tuning and RAG on knowledge-intensive tasks. It found that while fine-tuning offered some improvement, RAG consistently outperformed it, both for knowledge the model had seen in training and for entirely new knowledge, and that models struggle to learn new facts through unsupervised fine-tuning. That matches what teams see in practice: fine-tuning is a poor way to teach a model your price list or handbook, and a good way to teach it how to write.
How to choose: a five-question test
- Is the problem missing knowledge or wrong behaviour? Missing or outdated facts point to RAG. Wrong format, tone or task handling points to fine-tuning.
- How often does the content change? Weekly or daily changes rule out fine-tuning as the knowledge store.
- Must answers show their source? Legal, medical, financial and support answers usually must. That needs RAG.
- Do different users see different documents? Access control only works at retrieval.
- Have good prompts already failed on real cases? If not, fix the prompts first. If they have, test fine-tuning on the failing share.
Combining RAG and fine-tuning
The two are not exclusive. A fine-tuned model can be better at reading retrieved passages, following a citation format or using your terminology, while RAG keeps the facts current. Another common pattern uses fine-tuning to make a smaller model good at one narrow step, such as classifying requests, and RAG for the answer itself. In both, the knowledge stays in documents you control.
Data protection and security
- Erasure. GDPR Article 17 gives people the right to have their personal data erased without undue delay where a ground applies. Data in a RAG index can be deleted and re-indexed; data trained into model weights cannot be removed selectively, so keep personal data out of fine-tuning sets.
- Access. The OWASP Top 10 for LLM Applications (2025) lists vector and embedding weaknesses (LLM08), including unauthorised access and data leakage where access controls on embeddings are inadequate or misaligned. RAG must filter passages by the user's rights before the model sees them.
- AI literacy. Under AI Act Article 4, which has applied since 2 February 2025 and was replaced by Regulation (EU) 2026/1744, providers and deployers must take measures to support the development of AI literacy of their staff and others using the system on their behalf, without a required level for any individual.
Costs, without the numbers
RAG costs sit in preparing documents, building the index and search, testing answer quality and running a slightly longer request each time. Fine-tuning costs sit in collecting and cleaning examples, training runs, evaluation and repeating them when the data or base model changes. Fine-tuning a smaller model can lower running costs at high volume. Our guide to artificial intelligence cost explains what drives each part.
When to get a custom system
Ready-made AI search inside office suites works for general documents. A custom RAG system, with or without a tuned model, pays off when answers must come from your own systems, respect your permissions, cite sources and be measured against your own questions before launch.
Our RAG development service builds AI knowledge bases and assistants on your documents, with EU hosting and the code owned by you; generative AI development covers fine-tuning and wider AI features. For agents that act on the answers, see what is an AI agent, and for the budget of a support assistant, chatbot development cost. Every guide is in the guides library.
Questions people ask
What is the difference between RAG and fine-tuning?
RAG (retrieval-augmented generation) leaves the model unchanged and gives it relevant passages from your documents at the moment a question is asked. Fine-tuning changes the model itself by training it further on your examples. RAG adds knowledge that can be updated and cited; fine-tuning shapes behaviour, style and format.
Which is better, RAG or fine-tuning?
For answering questions from company knowledge, RAG is usually the better first choice: it is quicker to update, can show its sources and keeps access control on the documents. Fine-tuning is better when the model must reliably follow a format, tone or narrow task that prompts cannot achieve. Many systems use both.
Is RAG cheaper than fine-tuning?
Usually it is cheaper to start and to keep current, because updating knowledge means re-indexing documents, not retraining a model. Fine-tuning has training and evaluation costs each time the data changes, but it can reduce running costs when a smaller tuned model replaces a larger one. The total depends on volume and how often content changes.
Can you combine RAG and fine-tuning?
Yes. A common pattern is to fine-tune a model for the format, terminology or reasoning style of a task, and use RAG to supply the current facts. The fine-tuned model is then better at using the retrieved passages, while the knowledge itself stays in documents you can update and delete.
What about prompt engineering?
Prompt engineering, writing clear instructions and examples into the request, is the first step before either. Many projects that start by asking for fine-tuning reach their goal with good prompts plus RAG. Fine-tuning is worth testing when well-designed prompts still fail on a measurable share of real cases.
Sources
- Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401, 2020) (read 13 November 2026)
- Ovadia et al.: Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (arXiv 2312.05934, 2023) (read 13 November 2026)
- OWASP Top 10 for LLM Applications 2025: LLM08 Vector and Embedding Weaknesses (read 13 November 2026)
- Regulation (EU) 2016/679 (GDPR), Article 17 (read 13 November 2026)
- Regulation (EU) 2026/1744 amending the AI Act, new Article 4 (read 13 November 2026)