Most RAG systems start with vector search alone, because it is the default in every tutorial and it handles paraphrase well. Then someone searches for an error code, a part number or a clause reference, and the retriever returns passages that are about the right topic but never contain the string they typed.
Hybrid search and reranking are the two usual fixes, and they solve different problems. Hybrid search widens what you find by adding a keyword leg. Reranking reorders what you found with a slower, more careful model. This post covers how each one works, how results are combined, when a reranker earns its extra latency, and how to decide with an eval set rather than by instinct.
Keyword search and embeddings fail in different places
BM25, the default ranking function in Elasticsearch and OpenSearch, scores a passage by the query terms it contains. Rare terms count for more than common ones, repeated terms saturate instead of counting forever, and long passages do not win just by being long. It knows nothing about meaning: a search for "cancel my subscription" will not match a passage that only says "end your plan."
Embeddings turn the query and each passage into vectors and compare them by similarity, so paraphrase, synonyms and loose phrasing work well. The weakness is the mirror image. Identifiers such as SKUs, invoice numbers, error codes, version strings and legal section numbers carry little meaning a model can encode, so two codes that differ by one character can sit close together, and the exact one you need may not rank at all.
- Keyword search wins on exact identifiers, codes, names, acronyms, quoted phrases and domain jargon the embedding model rarely saw in training.
- Embedding search wins on paraphrase, on questions written in the user's words rather than the document's, and on conceptual queries such as "how do refunds work for annual plans."
- Both struggle when the answer is spread across several passages. That is a chunking and context problem, not a search one.
If your users mostly ask conceptual questions over well-written prose, embeddings alone may be enough, and that is the simpler system to run. If your corpus is full of codes, tables, product names or legal references, you almost certainly need the keyword leg. The store matters less than people expect, since most major engines now support both modes, a point we made in vector DB selection. One caution: Postgres's built-in text ranking is not BM25 (it lacks inverse document frequency, term saturation and length normalisation), so check what your keyword leg actually computes.
When should a RAG system use hybrid search and reranking?
Use hybrid search when your users search for exact strings, and add reranking when the right passages are found but ranked too low. Vector search handles paraphrase and loose wording, but embeddings blur identifiers such as SKUs, error codes, invoice numbers and legal section references. BM25 keyword search matches those exactly but knows nothing about meaning. Hybrid search runs both and merges the two lists, usually with reciprocal rank fusion, which scores each passage by its rank in each list and needs no tuning to start; weighted score fusion is an option once you have data to tune it. A cross-encoder reranker reads the query and each candidate together and reorders the shortlist more accurately, but it runs a model per candidate on every query, which adds latency. Decide with an eval set of real questions: compare recall@k with and without the keyword leg, and add a reranker only if recall at a larger k beats recall at the k you pass to the model.
How hybrid search combines two result lists
Running both searches is easy. The hard part is merging two ranked lists whose scores mean different things. A BM25 score has no fixed ceiling and depends on the query, while a cosine similarity sits in a narrow band. Adding them directly lets one leg drown out the other.
Reciprocal rank fusion, introduced in a 2009 paper by Cormack, Clarke and Büttcher, sidesteps the problem by ignoring scores entirely. Each passage receives 1 divided by (k plus its rank) from every list it appears in, and the sums decide the final order. A passage ranked highly by both legs rises to the top, and one found by only one leg still counts. The constant k, commonly 60 and the default in Elasticsearch, softens the advantage of first place. RRF needs no tuning to get started, which is why it is the usual default.
The alternative is score fusion: normalise each list's scores to a common range, then take a weighted combination. Weaviate's hybrid search defaults to a relative score fusion of this kind and exposes an alpha parameter that weights vector against keyword results. OpenSearch offers normalisation processors with configurable weights alongside rank-based fusion, and Qdrant offers both RRF and a distribution-based score fusion. Score fusion keeps more information, because it knows how far ahead the top result was, but the weights are one more thing to tune, and the right values can drift as your corpus changes.
- Start with RRF. It is hard to misconfigure and gives you an honest baseline.
- Try weighted score fusion when your eval set shows one leg is consistently more reliable for your traffic.
- Consider routing instead of fusing when queries split cleanly, for example sending anything shaped like an order number straight to an exact-match lookup.
- Retrieve more candidates from each leg than you will finally pass to the model, so fusion and reranking have something to work with.
What a cross-encoder reranker adds, and what it costs
Both retrieval legs are fast because they score the query and each passage separately: an index lookup, or a comparison against vectors computed ahead of time. A cross-encoder reads the query and a passage together, in one pass through the model, and outputs a single relevance score. Seeing both at once lets it notice that a passage uses the right term in the wrong context, or answers a slightly different question. The Sentence Transformers documentation describes the standard pattern: retrieve a candidate set with a fast method, then rescore only that set with the cross-encoder.
The cost is that none of this work can happen ahead of time. Every query runs the model once per candidate, which adds latency and compute to each request and grows with the size of the shortlist. Hosted reranking APIs move that load off your servers but add a network call and another vendor to the request path. A large language model can also rerank, by being asked to order or grade passages, with more flexibility and usually more latency again.
A reranker earns its place when:
- Your eval set shows the right passage is usually retrieved but ranked too low to reach the model, the pattern where recall improves clearly as k grows.
- You can only pass a few passages to the model, because of context cost, latency or answer quality, so the order at the top matters a great deal.
- Queries are long or nuanced, and being on the right topic is not the same as answering the question.
- Your latency budget has room for another model call, measured on your own hardware and shortlist size rather than taken from someone else's benchmark.
It is not worth it when retrieval misses the right passage entirely. A reranker can only reorder what it is given. If the answer never makes the shortlist, the fix is upstream in chunking, embeddings or the keyword leg, which is one of the RAG anti-patterns we keep finding.
Decide with an eval set, not a demo
Every choice above should be settled by measurement on your own questions. Build the set from real queries, label the passages that answer them, and include the hard cases on purpose: exact codes, paraphrased questions, multi-part questions and questions the corpus cannot answer. The full method is in our guide to RAG evaluation metrics.
- Measure recall@k for vector search alone, at the k your pipeline actually uses and at a larger one.
- Add the keyword leg with RRF and measure again, broken down by query type. Identifier queries are usually where the difference shows first.
- If recall at the larger k is clearly better than at your real k, add a reranker over the larger set and measure precision and MRR at the real k.
- Measure latency at each stage under realistic load, looking at the slow tail as well as the median, since the slow requests are the ones users notice.
- Check end-to-end answer quality, faithfulness and citations, not only retrieval scores, before you ship the change.
- Keep the configuration in version control and rerun the suite on every index rebuild or model change.
Read the per-query view as well as the averages. A change that helps conceptual questions and hurts code lookups can look neutral overall while breaking the queries your support team cares about most.
Apply permissions inside each leg, not after
Whatever combination you choose, apply access filters inside each retrieval leg, before fusion and reranking. Filtering after the reranker means restricted passages get scored, can crowd permitted ones out of the shortlist, and are one bug away from reaching the model. The details are in access control for RAG.
Start simple, and add each layer when the numbers ask for it
A sensible order for most teams is a keyword leg with RRF from the start, since it is cheap to add and protects exact-match queries; a reranker only when the eval set shows ranking, not recall, is the problem; and weighted fusion or query routing once you have enough labelled traffic to tune them. Each layer is another moving part to monitor, so add it because a measurement asked for it, not because a diagram had it.
This is the retrieval layer we build in RAG development, starting with the eval set and adding hybrid search and reranking where they help on your questions. Realty One RAG, a conversational property search, combines both. If your assistant answers confidently about the wrong part number, Octalcode can help you find which layer is missing.