Retrieval-augmented generation is the most common way teams put a language model to work on their own data, and it is also the most common place we are called in to fix one that is not working. The pattern is simple on a whiteboard: retrieve relevant documents, hand them to a model, get a grounded answer. In production it fails in remarkably consistent ways. Here are the six we find most often, and how each one is usually fixed in less than a day once you know to look for it.
One: treating retrieval quality as the model’s problem
When a RAG system gives a wrong or vague answer, the instinct is to blame the model and reach for a bigger one. Most of the time the model never saw the right information. If the retrieval step did not surface the relevant chunk, no amount of model intelligence will conjure it. The failure is upstream.
Before touching prompts or swapping models, measure retrieval in isolation. For a set of real questions, check whether the correct source chunk appears in the top results at all. If your retrieval recall is low, that is your entire problem, and a better model would just be a more expensive way to be confidently wrong.
Two: chunking by character count and hoping
The default tutorial splits documents into fixed windows of a few hundred characters with some overlap. It is easy and it quietly destroys meaning. A definition gets severed from its term, a table is cut in half, a clause loses the sentence that qualified it. The retriever then matches fragments that no longer say what they meant.
Chunk along the structure the document already has. Split on headings, sections, list items, and natural boundaries. Keep a chunk semantically whole even if that makes it longer or shorter than a round number. Attach the section heading and document title to each chunk as context. This single change moves more answers from wrong to right than any model upgrade we have made.
If the retrieval step did not surface the relevant chunk, no amount of model intelligence will conjure it.
Three: pure vector search with no keyword fallback
Embedding search is excellent at meaning and surprisingly poor at exact terms. Product codes, error numbers, names, and acronyms are exactly where users expect precision and exactly where vector similarity drifts. Someone searches for a specific part number and gets semantically adjacent paragraphs that never mention it.
Run hybrid retrieval: combine vector similarity with a classic keyword index and merge the results. Hybrid search is now a near-default for serious systems because it covers both failure modes at once. The fix is rarely more than adding the keyword leg and a small reranking step on top.
Four: shipping without an evaluation set
This is the anti-pattern underneath all the others. Teams tune RAG by trying a change, eyeballing a few answers, and deciding it feels better. Then a later change quietly breaks what the first one fixed, and nobody notices until a user complains. Without measurement you are not engineering, you are decorating.
Build a small evaluation set of real questions with known good answers, fifty is enough to start. Score retrieval recall and answer quality on every change. The moment you can see a number move, tuning stops being guesswork and starts being engineering. We attach an eval harness to every RAG build before we tune anything, because it is the only way to know a change helped.
Five: no citations, so no trust and no debugging
A RAG answer with no source attached is a liability twice over. Users cannot verify it, so they either trust it blindly or not at all, and when it is wrong you have no way to tell whether retrieval failed or the model ignored what it was given. You are debugging in the dark.
Make the system return the chunks it used alongside every answer, and surface them in the interface as citations. This builds user trust, turns every wrong answer into a debuggable case, and gently constrains the model toward grounding its response in the retrieved text rather than its own memory.
Six: a static index in a world that changes
Many RAG systems are indexed once at launch and then slowly rot. Documents change, policies update, products are added, and the index keeps confidently serving last quarter’s truth. The model is not hallucinating, it is faithfully retrieving stale facts, which is harder to spot and easier to trust.
Treat the index as a living pipeline, not a one-time load. Re-index on a schedule or, better, when source documents change, and stamp each chunk with a date so you can reason about freshness. For anything where correctness matters, knowing how old a retrieved fact is should be part of the answer.
The common thread
Every one of these failures is a measurement gap before it is a technology gap. Teams cannot see that retrieval is missing, that chunks are broken, that an index is stale, or that a change made things worse, because nothing is being measured. The fix is almost always to make the system observable first, then improve the part the numbers point to.
That is the order we work in: instrument, measure, then improve the thing that is actually broken. If you have a RAG system that demos well but disappoints in production, the diagnosis usually takes an afternoon and the fixes are rarely exotic. We are happy to take a look.