Most RAG systems are evaluated the same way before launch: someone asks a handful of questions, the answers look good, and the team ships. Weeks later users report answers that are confident, fluent and wrong, and nobody can tell whether retrieval missed the right document or the model ignored it.
RAG evaluation metrics fix that, but only if you measure the two halves of the system separately and against a test set built from real questions. This is the approach we use before any RAG system goes live, and it follows directly from the failures we described in six RAG anti-patterns we keep finding in production.
Measure retrieval and generation separately
A RAG pipeline has two jobs. Retrieval finds the passages that contain the answer. Generation turns those passages into a correct, relevant response. A single end-to-end score blends the two together, so when it drops you do not know which half to fix.
Split the scores. If retrieval is missing the right passage, no prompt change will help, and the fix lives in chunking, embeddings, hybrid search or reranking. If retrieval is fine and the answer is still wrong, the problem is in the prompt, the model or how much context you pass. Separate numbers turn a vague quality complaint into a specific engineering task.
Build the test set before you tune anything
The test set is the foundation, and it is where most teams cut corners. Build it from real questions: search logs, support tickets, and the questions subject experts get asked every week. Questions invented by the people who built the index tend to reuse the wording of the documents, which makes retrieval look better than it is.
- For each question, record the source documents or passages that contain the answer, plus a short reference answer written or approved by someone who knows the domain.
- Label relevance at a level that survives re-chunking, such as document plus section, so changing your chunk size does not invalidate the whole set.
- Include hard cases on purpose: exact identifiers and codes, questions that need two passages combined, ambiguous phrasing and outdated terminology.
- Include questions the corpus cannot answer, so you can check that the system says so instead of inventing something.
- Keep the set in version control and add to it every time a real failure shows up.
Breadth across question types matters more than raw size at the start. A modest set that covers your real traffic will show you more than a large one generated from the documents themselves. This is the same habit we apply to every SI project, described in building the eval harness before the model.
Retrieval metrics: recall@k, precision@k and MRR
Retrieval metrics compare the ranked list your retriever returns with the passages you labelled as relevant. Here k is the number of results you pass to the model, and it should match what your pipeline actually uses, not a number chosen to flatter the score.
- Recall@k asks how much of the relevant material appears in the top k results. If a question needs two passages and only one is retrieved, recall for that question is half. For RAG this is usually the metric to watch first, because the model cannot use what it never sees.
- Precision@k asks how much of the top k is actually relevant. Low precision means the model is reading noise alongside the answer, which costs tokens and can pull the response off course.
- MRR, mean reciprocal rank, looks at where the first relevant result appears. A relevant passage in first place scores full marks and one further down scores less. It tells you whether your ranking puts the best evidence first.
- nDCG extends the same idea to graded relevance, rewarding a ranking that puts the most useful passages highest. It is worth adding once you label passages as highly or partly relevant rather than a plain yes or no.
Look at these at more than one value of k. If recall is poor at a small k but good at a larger one, the right passages are being found but ranked too low, and a reranker will often help. If recall is poor even at a large k, the problem is upstream in chunking, embeddings or the absence of keyword search.
Answer metrics: faithfulness, relevance and citations
Once retrieval is measured, score the answers. Three metrics cover most of what matters.
- Faithfulness, also called groundedness, asks whether every claim in the answer is supported by the retrieved passages. It is usually measured by splitting the answer into individual claims and checking each one against the context.
- Answer relevance asks whether the response addresses the question that was asked, completely and without wandering. A faithful answer to a different question still fails.
- Citation correctness asks whether each cited source actually supports the sentence it is attached to. Users trust citations, so a wrong citation does more damage than a missing one.
Faithfulness and correctness are not the same thing. An answer can be correct and unfaithful if the model filled a gap from its own memory, and that is a warning sign even when it happens to be right, because next time the memory will be stale or wrong. Add one more check for the unanswerable questions in your set: did the system decline, or say what it could not find? Refusing correctly is a feature, and it is easy to break while tuning for helpfulness.
An answer can be correct and unfaithful if the model filled a gap from its own memory.
LLM-as-judge: useful, with caveats
Most answer metrics need a model to grade them, because faithfulness and relevance cannot be checked with string matching. Using a language model as a judge is practical and scales well, but a judge is another model with its own failure modes.
- Judges have known biases. They tend to prefer longer answers, can favour whichever option they read first in a comparison, and may rate output from their own model family more kindly.
- Vague scales produce noisy scores. Ask narrow yes-or-no questions per claim or per criterion rather than asking for a broad numeric rating.
- Pin the judge model and its prompt, and version them like code. Changing the judge changes every score, which can look like a regression or an improvement that never happened.
- Calibrate against people. Have a domain expert label a sample, compare their labels with the judge, and keep spot-checking the cases where the two disagree.
Treat judge scores as a strong signal for comparing versions of your own system, not as an absolute measure of truth.
Run the regression on every change
The point of all this is a suite you run every time something changes, not a report you produce once before launch. Chunking strategy, embedding model, reranker, prompt, generation model, the number of passages passed in and every index rebuild can all move quality, sometimes in opposite directions for different kinds of question.
Wire the suite into your release process. Compare each run with the last accepted baseline, read the per-question view as well as the averages, and block a release when recall or faithfulness drops on the question types you care about. That is how the engineering-ops copilots with a custom eval harness we built are run: regression suites execute on every model change, so a swap is a measurement rather than a gamble. Setting up and maintaining that kind of harness is the core of our SI-driven QA and evaluation work.
Where to start this week
Pull real questions from your logs, label the passages that answer them, and run recall@k on your current retriever before touching anything else. That one number usually tells you where the next month of work should go. Then add faithfulness and citation checks, calibrate the judge against a human sample, and put the whole suite in front of every change.
If your RAG system goes live soon and the only evidence it works is a few good demos, the test set is the place to begin. If you want help building the metrics and the regression suite around it, our SI-driven QA team does exactly that, before launch and on every change after it.