HOW HYBRID SEARCH WORKS
In 2023–2024, the canonical RAG looked like this: split documents into chunks, ran them through an embedding model, stored them in a vector DB, retrieved top-k by cosine similarity on query, and shoved them into the prompt. It works — until you start looking at what actually ends up in the context.
Vector (dense) search excellently captures meaning, but systematically misses where literalness matters: article numbers, error codes, function names, legal terms, rare abbreviations. The query «error E4102 during deployment» gets dissolved by the embedding model into a «general cloud about deployment», and the needed chunk with the exact code drops to 40th place. The user gets a confident but wrong answer. This is the main source of hallucinations — not the LLM itself, but garbage in the input.
From this follow two orthogonal techniques that became standard in 2026: hybrid search (we combine lexical and semantic signals) and reranking (we reorder candidates with a model that sees the query and document together).
BM25 IS NOT GOING ANYWHERE
BM25 — is a lexical ranker based on TF-IDF with frequency saturation and length normalization. It’s almost 30 years old and still beats neural networks on exact matches because it simply counts token overlap. Zero training, zero GPU, milliseconds per query.
BM25’s weakness — zero understanding of synonyms and paraphrase. The query «how to speed up model training» won’t find a chunk about «optimizing neural network training» if there are no common words. Exactly where dense search shines.
That’s why they aren’t opposed but combined. Two retrievers, two candidate lists, one unified ranking. Fresh releases of retrievers and re-rankers are easy to track in the REDDYX catalog — models are updated literally every week.
HOW HYBRID SEARCH IS BUILT
Hybrid search runs the query through both channels and merges the results. The main question — how to combine two scores of different nature: BM25 yields unbounded positive numbers, cosine ranges from -1 to 1. You can’t compare them directly.
Reciprocal Rank Fusion (RRF)
The most robust way in 2026 is RRF. It ignores absolute score values and works only with positions in the lists. The formula is simple:
score(d) = Σ 1 / (k + rank_i(d))
# rank_i — позиция документа d в i-м списке (1-based)
# k — константа сглаживания, обычно 60
RRF requires no calibration or normalization, is robust to outliers, and that’s why it’s used by default in Elasticsearch, OpenSearch, Weaviate, and Qdrant. The alternative — a weighted sum of normalized scores (min-max or z-score) — is finicky about distribution and requires tuning weights for the domain.
RERANKING: THE MAIN LEVER OF QUALITY
Hybrid search gives a good candidate list, but it’s still fast and «crude»: a bi-encoder encodes query and document separately, losing fine-grained interactions between them. A reranker is a cross-encoder: it feeds the (query, document) pair into the model simultaneously, and attention sees them together. Accuracy is an order of magnitude higher, but so is the cost: you have to compute it for each pair separately, so the reranker processes only the top‑50–100 candidates from the first stage, not the whole corpus.
The two-stage scheme:
- Retrieval (recall stage): hybrid search retrieves 50–100 candidates. The goal — not to lose relevant items, precision is secondary.
- Rerank (precision stage): the cross‑encoder reorders them and leaves the top‑3–8 for the LLM context.
According to community measurements, adding a reranking stage on top of plain vector search gives the biggest quality boost among all «cheap» RAG improvements — noticeably higher than switching the embedding model or tuning chunk size. This is the first thing to implement.
WORKING EXAMPLE IN PYTHON
Minimal pipeline: BM25 (rank-bm25) + dense (sentence-transformers) → RRF → cross-encoder reranker. No external services, everything locally.
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, CrossEncoder, util
import numpy as np
docs = [...] # список чанков (строки)
tokenized = [d.lower().split() for d in docs]
# --- Этап 0: индексы ---
bm25 = BM25Okapi(tokenized)
embedder = SentenceTransformer("intfloat/multilingual-e5-large")
doc_emb = embedder.encode(docs, normalize_embeddings=True)
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")
def rrf(ranked_lists, k=60, top=100):
scores = {}
for lst in ranked_lists:
for rank, doc_id in enumerate(lst, start=1):
scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
return sorted(scores, key=scores.get, reverse=True)[:top]
def search(query, top_k=5):
# лексический список
bm25_scores = bm25.get_scores(query.lower().split())
bm25_rank = np.argsort(bm25_scores)[::-1][:100]
# семантический список
q_emb = embedder.encode(query, normalize_embeddings=True)
dense_scores = util.dot_score(q_emb, doc_emb)[0].numpy()
dense_rank = np.argsort(dense_scores)[::-1][:100]
# гибрид через RRF
candidates = rrf([list(bm25_rank), list(dense_rank)], top=50)
# reranking кросс-энкодером
pairs = [(query, docs[i]) for i in candidates]
rr = reranker.predict(pairs)
order = np.argsort(rr)[::-1][:top_k]
return [docs[candidates[i]] for i in order]
print(search("ошибка E4102 при деплое"))
Key detail: for e5 models in real production, add prefixes query: and passage: — without them embedding quality drops. And keep the reranker on top‑50, not on the whole output: this balances latency and accuracy.
WHAT TO CHOOSE: COMPARISON OF APPROACHES
| Approach | Top‑k quality | Latency | Cost | When to take |
|---|---|---|---|---|
| Only BM25 | Low on paraphrase | ~1–5 ms | Penny‑cheap | Code, logs, exact terms, MVP without GPU |
| Only dense | Medium | ~10–30 ms | Medium | Semantic queries, chat‑bot FAQs |
| Hybrid (BM25+dense, RRF) | Good | ~15–40 ms | Medium | Default for most RAG |
| Hybrid + reranker | High | +50–300 ms | Higher (GPU/API) | Prod, where answer accuracy matters |
| Hybrid + LLM-as-reranker | Maximum | seconds | Expensive | Legal/Medicine, low traffic |
Practical rule 2026: almost everyone needs a «hybrid + cross-encoder-reranker». Leave LLM‑reranking for narrow domains with high cost of error and low QPS.
METRICS: HOW TO KNOW IT GOT BETTER
«By eye it got better» — not an argument. Collect a golden set of 50–200 pairs «query → relevant chunks» and measure retrieval separately from generation:
- Recall@k — proportion of relevant documents that made it into top‑k. Main metric of the recall stage.
- nDCG@k — accounts for position: a relevant item at position 1 is worth more than at position 10. Main metric of the reranking stage.
- MRR — how highly the first relevant document is ranked.
- Faithfulness / groundedness — already at the LLM level: does the answer rely on the provided context. It is measured via RAGAS or an LLM judge.
Separation is critical: if retrieval metrics are good but answers are bad, the issue is with the prompt or model, not the search. Don’t tune everything blindly at once.
COMMON PITFALLS
- Reranker on top of garbage. If the recall stage didn’t retrieve a relevant document into the top‑100, the reranker can’t resurrect it. First fix recall.
- Chunks too small. On chunks of 1–2 sentences context is lost; on overly large chunks the reranker smears the signal. A sensible start is 256–512 tokens with overlap.
- Ignoring language. For Russian use multilingual models (e5, bge-m3), and run BM25 through normalization/lemmatization — otherwise cases kill the lexical match.
- Reranking the whole database. A cross‑encoder on 10k documents takes seconds and GPU cost. Only top‑50–100.
- No offline evaluation. Without a golden set any «improvement» is faith, not engineering.
Frequently asked questions
What is reranking in RAG in simple terms?
Reranking is the second stage of search, where a cross‑encoder model re‑orders the already retrieved candidates, evaluating the pair «query + document» together. This is more accurate than plain vector search and boosts the relevance of the top documents that go into the LLM context.
Is hybrid search always better than vector?
Almost always. Hybrid search combines the BM25 lexical signal (exact words, codes, terms) and the semantics of embeddings (meaning, synonyms), covering the weaknesses of each. It loses to pure dense only on rare domains without exact terms, where such scenario occurs rarely.
Is reranking needed if I have a good embedding model?
Yes. Even a strong bi-encoder encodes the query and document separately and loses fine-grained connections between them. A cross-encoder-reranker sees them simultaneously and usually gives a greater quality boost than changing the embedding model.
How much does reranking slow down the response?
A local cross-encoder on the top-50 candidates adds roughly 50–300 ms on GPU; API re-rankers are comparable plus network round-trip. For most production RAG, this is acceptable, because the accuracy gain reduces the number of user's repeat queries.