Pandu: Self-Hostable RAG, Measured End to End

The problem

Anyone can wire a vector store to an LLM in an afternoon. The difference between that demo and infrastructure is everything around the pipeline: retrieval quality that is measured rather than felt, independence from any vendor, observability, and evaluation that runs in CI — not once, at demo time. Pandu (Indonesian for “guide”) was built to show that difference: every architectural claim in its README is enforced by a CI gate, not merely asserted.

What we built

A self-hostable RAG platform: upload documents, ask questions, get streamed answers with source citations. The backend is a FastAPI modular monolith with import-linter-enforced module boundaries; a single Postgres holds relational data, vectors (pgvector), and lexical search (FTS) at once. A question is embedded and searched through two arms in parallel — dense (HNSW) and lexical — fused with Reciprocal Rank Fusion, then the top five contexts enter a grounded prompt with numbered [1][2] citations that are verified in code, not requested in a prompt.

Chat, embeddings, and reranking are three separate ports, so adding a vendor means one adapter file. Observability runs through self-hosted Langfuse (ClickHouse) recording model ids, counts, and latencies — prompt and document text never leave the app.

Engineering decisions

No RAG framework, no gateway SDK. The retrieval pipeline is hand-implemented so every part of it can be measured, and the seam that all API keys flow through is owned outright.

Thresholds ratchet. The CI gates (recall@5 ≥ 0.60; faithfulness ≥ 0.85) may be raised, never lowered, and --compare fails a run that regresses against the previous stored run even when every absolute threshold passes — a mean can lie; a diff does not.

Abstention is measured, not prompted. The golden set carries 5 negative questions the corpus genuinely cannot answer, and both error directions are reported separately.

The measured results

On the NIST corpus (golden_v2, 2026-09-06): recall@5 0.967, MRR 0.947, nDCG@5 0.920, abstention recall 1.000. More valuable than the numbers is the story behind them: a configuration sweep found the lexical arm silently empty on 14 of 20 questions — “hybrid” had been running dense-only for three weeks with every test green, caught only when per-arm provenance was read on a single question. After the fix, hybrid actually beats dense (nDCG@1 0.933 vs 0.800).

And one gate is deliberately left red: LLM-judge faithfulness at 0.810 against the 0.85 gate, with a measured diagnosis that cross-document questions (0.543) are the defect. The “obviously right” fix — per-document context caps — was measured and then rejected with numbers, not shipped on the strength of a plausible hypothesis. That is the culture this repo is selling.

Running it

git clone https://github.com/ridwanspace/pandu && cd pandu && make up — the full stack (web :3000, API :8000, Postgres, Redis) via Docker Compose. The demo needs no API key: deterministic mock providers run the entire pipeline and are clearly labelled mock/extractive; add one provider key when you want real generation quality.