Hybrid RAG Reference — Benchmarkable Hybrid Retrieval

⚠ This is an open-source project, not client work. The benchmark dataset is synthetic (288 documents, 120 labelled queries), the mock numbers characterise the pipeline’s plumbing, and the README puts a reproducing command next to every number — including the section that proves its own early conclusions wrong.

The problem

Most advice about retrieval for RAG is folklore: “use hybrid”, “add a reranker”, “chunk at 512 tokens”. Rarely does anyone measure whether those stages actually help on your corpus. This repo is a hybrid retrieval implementation you can benchmark, not another chatbot demo — every stage is optional and independently benchmarkable, so “BM25 alone” and “hybrid + rerank” are the same object with different stages switched on, which makes the comparison apples-to-apples.

What was built

Okapi BM25 over an inverted index (implemented here, not imported), dense vector search over numpy or pgvector behind one interface, RRF fusion plus min-max and z-score normalisation, optional cross-encoder reranking, and query transformations (multi-query, HyDE). The evaluation harness measures recall@k, nDCG@k, MRR, hit-rate@k, and per-stage latency percentiles.

The bundled dataset is generated from a fixed seed and regenerates byte-identically: 288 DevOps handbook documents and 120 labelled queries in three styles — lexical (reuses the document’s wording), paraphrase (vocabulary the document never uses), and multi-hop (several documents all relevant). Labels are exact by construction, and because each query style is labelled, the harness can separate them — which is where the actual findings live.

Findings, including the ones that did not work

On the mock run (deterministic hash embedder, fully offline): the reranker was the biggest disappointment — hybrid + rerank lowered nDCG@10 from 0.518 to 0.442. The cause is measured: the mock reranker produces a mean of 6.8 distinct scores across 50 candidates, so most documents tie and tie-breaking discards RRF’s fine-grained ordering. A reranker that cannot separate its candidates is worse than no reranker. Yet the same reranker was the single best technique on the paraphrase slice (recall@10 0.417 → 0.500) — the aggregate table hides all three effects. Chunking also mattered more than fusion: dropping from 200-word to 80-word chunks cost more recall@10 (0.658 → 0.512) than any fusion choice gained.

Real embeddings change the answer

The same benchmark run against gemini-embedding-001 (768 dimensions, measured 2026-08-16, ~25 minutes wall-clock): dense alone jumps from losing to BM25 (nDCG@10 0.496 vs 0.537) to winning decisively (0.855), and hybrid RRF — the best configuration in the mock table — becomes worse than dense alone (0.701). The mechanism is the same: RRF is rank-based and weights its inputs equally, so fusing a strong retriever with a much weaker one drags the strong one down. Three of this repo’s own mock-based conclusions do not survive contact with a real embedder — and the README says that is the main reason the section exists.

One more live experiment: HyDE with pseudo-documents generated by deepseek-v4-flash (120 queries, 123 live calls) lifted recall@10 from 0.658 to 0.703 — the best of any configuration — while dropping nDCG@10 to 0.472, the worst. Reporting only “HyDE improved recall by 4.5 points” would be true and misleading.

Run it yourself

make install
make bench    # full benchmark on the bundled dataset, offline
make test     # unit tests: no network, no keys, no database

The pgvector backend (with an HNSW index) is one docker compose up away, and the loader reads any JSONL corpus/queries pair or your own BEIR-format dataset. Public repo: github.com/ridwanspace/hybrid-rag-reference.