Almost every team that wants a language model to "know our stuff" asks the same question early: should we use retrieval-augmented generation or fine-tune a model on our data? It sounds like a technology choice. It is really a question about what problem you have, and most teams answer it before they have looked closely at the problem.
Here is the framework we use with clients. It fits on one page, it rarely takes more than a meeting to apply, and it usually ends with a clearer answer than the vendor slide that prompted the question.
The one distinction that decides most cases
RAG changes what the model knows at the moment it answers. Fine-tuning changes how the model behaves every time it answers. Retrieval puts the right documents in front of the model for each question, so the knowledge lives outside the model in an index you control. Fine-tuning adjusts the model’s weights using examples, so the change lives inside the model.
That gives you the first and most useful test. If your problem is "the model does not know our policies, products, contracts or tickets", it is a knowledge problem, and knowledge problems are retrieval problems. If your problem is "the model knows enough but answers in the wrong format, tone or structure, or keeps failing at a narrow task we can show it many examples of", it is a behaviour problem, and behaviour is where fine-tuning earns its place.
Fine-tuning teaches a model how to answer. Retrieval tells it what is true today. Confusing the two is the most expensive mistake in this decision.
Teams get this wrong in a predictable direction. They fine-tune on their documentation hoping the model will memorise it, then discover that it recalls facts unreliably, blends them with what it learned in pretraining, and cannot tell anyone where an answer came from. Fine-tuning is a poor way to store facts.
When RAG wins
Retrieval is our default starting point for any product that answers questions about a body of content. It wins clearly when one or more of these is true:
- Your data changes. Inventory, policies, listings and tickets move weekly or daily. Updating a RAG system means re-indexing a document, not retraining a model.
- Answers need citations. Retrieval knows which chunks it used, so the interface can show sources and users can check them.
- Different users may see different data. Retrieval can filter by permission at query time. A fine-tuned model cannot forget what it learned for one user while answering another.
- Data must be removable. When a document is withdrawn or a deletion request arrives, you remove it from the index. Removing it from model weights means retraining.
- You need to debug wrong answers. A bad answer traces back to either a retrieval miss or a generation failure, which is the difference between a fix and a guess.
Our Realty One property-search copilot is a typical case. Listings change constantly, every answer has to be grounded in real listings, and inline citations are what make users trust it. No amount of fine-tuning would have kept a model current with that catalogue. Hybrid retrieval with re-ranking did.
RAG has its own failure modes, and most of them sit upstream of the model: careless chunking, pure vector search that misses exact terms, indexes that quietly go stale. We wrote up the ones we see most in RAG in production: six anti-patterns, and they are worth ruling out before you conclude that retrieval "does not work" for your data.
When fine-tuning wins
Fine-tuning is the right tool when the model’s knowledge is fine but its behaviour is not, and better prompting has stopped helping. These are the cases where we recommend it:
- Strict output formats. The model must produce a specific structure, house style or domain notation every time, and prompts with examples get it right often but not reliably enough.
- Narrow, high-volume tasks. Classification, extraction or routing where you have plenty of labelled examples and want a smaller, faster model to do one job well.
- Tone and voice that are hard to describe. Some styles are easier to show than to explain, and good examples teach them better than a long system prompt.
- Serving efficiency. A fine-tuned small model can replace a large general model on a narrow task, which makes it faster and lighter to run at volume.
- Teaching a skill rather than a fact. Following a specific procedure, using your internal tool schema consistently, or handling domain jargon the way your experts do.
Notice what is missing from that list: "learning our documents". Fine-tuning can nudge a model toward your vocabulary, but if the goal is accurate, current, attributable answers about your content, it is the wrong tool on its own.
The cost of change, which teams rarely plan for
This decision is not only about what works on launch day. It is about every day after, when the data changes, a provider ships a better model, or a requirement shifts under you.
With RAG, most changes are small and local. A new document is an index update. A better base model is a configuration change plus an evaluation run. A new access rule is a filter. The model stays swappable because nothing about your business is baked into it.
With fine-tuning, change is heavier. New knowledge means a new training run. A new base model means repeating your training on it and validating again, because a fine-tune belongs to the model it was trained on. Hosted providers also retire older models over time, and a fine-tune built on a retired model has to be rebuilt. The training dataset becomes an asset you maintain: versioned, cleaned and reviewed. None of this is a reason to avoid fine-tuning. It is a reason to be sure the behaviour gain is worth carrying.
Data freshness and the shape of your data
Two quick questions narrow the choice further. How often does the information change, and does it look like documents or like examples?
Information that changes faster than you would retrain a model belongs in retrieval. That covers almost all operational data: catalogues, listings, tickets, policies, account records. Information that is stable and naturally expressed as input and output pairs (this email became this category, this messy form became these fields) is training data, and fine-tuning can use it directly.
There is also a third option that teams skip. For a small, stable corpus, a long-context model can take the whole thing in the prompt. If your knowledge base fits comfortably, that can be the simplest system to run. It stops working well once the corpus grows, or once you need permissions and citations at the level of individual passages.
When to combine them
The mature answer for many products is both, each doing the job it is good at. Retrieval supplies current, permissioned, citable knowledge. A fine-tuned model handles behaviour: answering in the exact format the interface needs, using your terminology, or calling tools consistently.
The order matters. Start with retrieval and a strong general model, because that combination reaches a working product fastest and keeps every option open. Most of the engineering in a production RAG system goes into the retrieval side anyway: chunking, hybrid search, re-ranking and freshness. Add fine-tuning only when your evaluation set shows a specific, persistent behaviour gap that better prompts and examples have not closed. Fine-tune first and you lose the ability to tell whether a bad answer came from the training, the retrieval or the prompt.
One combined pattern we like: fine-tune the model on examples of answering from retrieved context, so it learns to stay grounded, cite properly and say "I don’t know" when the retrieved text does not cover the question. That teaches behaviour, which is what fine-tuning is for, while the knowledge stays in the index where you can update it.
A decision checklist
- Is the gap knowledge or behaviour? Knowledge points to RAG, behaviour points to fine-tuning.
- Does the data change faster than you would retrain? If yes, retrieval.
- Do users need sources, or do permissions differ between users? If yes, retrieval.
- Have you exhausted prompting with good examples? If not, do that before any fine-tune.
- Do you have a clean, representative set of examples of the behaviour you want? If not, you are not ready to fine-tune.
- Can you measure the difference? If you cannot score both approaches on the same evaluation set, you are choosing on opinion.
Measure before you commit
The last question is the one that settles arguments. Build a small evaluation set of real questions with known good answers, then run each candidate approach against it. We set this up before any model work, as described in our eval-first approach, because it turns "which is better" from a debate into a number you can read.
In practice the harness usually shows that retrieval quality is the bottleneck long before model behaviour is. That is why most of our engagements start with retrieval, and many never need a fine-tune at all.
If you are weighing this decision for a product right now, we are happy to look at your data and your failure cases and tell you which side of the line you are on. It is often the first conversation in our RAG development work, and it is usually a short one.