Hybrid Search & Reranking

Pure cosine-similarity retrieval will carry you surprisingly far — and then it will fail on exactly the queries your users care about most: product codes, error messages, people's names, and questions where "semantically similar" isn't the same as "actually relevant." This lesson covers the three upgrades every serious RAG system eventually adopts: hybrid search, reranking, and metadata filtering.

The problem: two kinds of matching

Consider these real query types against a support knowledge base:

• "How do I get my money back?" — zero word overlap with "Refund policy," but obviously about it. Semantic search wins.
• "What does ERR_4102 mean?" — the exact token ERR_4102 appears in exactly one doc. Keyword search wins.
• "Refund policy for Enterprise plan customers in the EU" — needs semantic matching and hard constraints (plan, region). Neither alone is enough.

Vector search treats text as geometry; keyword search treats it as an inverted index of exact terms (the classic algorithm is BM25, a smarter cousin of TF-IDF that rewards rare terms and normalizes for document length). Their failure modes are almost perfectly complementary — which is the whole opportunity.

Hybrid search: run both, fuse the results

Hybrid search runs vector search and BM25 in parallel, then merges the two ranked lists. The most popular fusion method, Reciprocal Rank Fusion (RRF), is delightfully dumb — it ignores the raw scores entirely and only uses each document's rank in each list:

def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    """rankings: e.g. [vector_results, bm25_results], best first."""
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

A document ranked #1 by one system and #3 by the other beats a document ranked #2 by only one. Because RRF never compares raw scores, it sidesteps the annoying fact that cosine similarities and BM25 scores live on completely different scales.

Reranking: a second, smarter opinion

Embedding search is a bi-encoder: the query and each chunk are embedded separately, and relevance is approximated by vector distance. Fast — you can scan millions of precomputed vectors — but the model never actually reads the query and chunk together.

A reranker is a cross-encoder: it takes the pair (query, chunk) as one input and outputs a relevance score, attending across both texts at once. Far more accurate, far too slow to run on a million chunks. So you use the classic two-stage funnel:

1. Fast retrieval (hybrid) fetches ~25-100 candidates.
2. The reranker scores those candidates precisely.
3. The top 3-5 go into the prompt.

candidates = hybrid_search(query, top_k=50)          # fast, rough
scored = reranker.score([(query, c.text) for c in candidates])
best = [c for c, _ in sorted(zip(candidates, scored),
                             key=lambda p: p[1], reverse=True)[:5]]

In practice, adding a reranker is often the single largest retrieval-quality win available — typically worth more than switching embedding models — because it directly attacks the "prompt dilution" failure from the previous lesson.

Metadata filters: constraints aren't semantic

Some things should never be fuzzy. "Documents for the Enterprise plan," "only pages updated after March," "only this user's tenant" — these are hard predicates, and you enforce them as filters on the chunk metadata you (wisely) stored at ingestion:

results = vector_db.search(
    vector=embed(query),
    filter={"plan": "enterprise", "region": "eu",
            "updated_at": {"$gte": "2025-03-01"}},
    top_k=25,
)

Filters run inside the search (pre- or post-filtering depending on the database), and they're also your security boundary: multi-tenant RAG without a tenant filter is a data breach with extra steps.

Build it yourself

1. Add BM25 to your mini-RAG from the last lesson (the rank_bm25 Python package is a 5-line integration).
2. Implement rrf() as above and fuse the two result lists.
3. Craft two test queries: one paraphrase-style ("get my money back") and one exact-identifier-style (an error code or function name). Compare vector-only, BM25-only, and hybrid rankings side by side.
4. Bonus: run a cross-encoder from sentence-transformers (e.g. cross-encoder/ms-marco-MiniLM-L-6-v2) over your top-20 and see how the order changes.

Summary

• Vector and keyword (BM25) search have complementary blind spots; hybrid search runs both and fuses rankings, typically with RRF.
• RRF fuses by rank position, dodging the incomparable-scores problem.
• Rerankers (cross-encoders) read query and chunk together for precise scoring — used as a second stage over ~25-100 fast-retrieved candidates, and often the biggest quality win available.
• Hard constraints (tenant, plan, date, permissions) belong in metadata filters, never in prompt instructions.
• The production retrieval funnel: hybrid retrieve wide → rerank → top 3-5 chunks into the prompt.