RAG that actually works: lessons from 30 production pipelines
Reranking, hybrid search, eval harnesses and the unsexy plumbing that separates demos from systems.
Naive RAG — embed the docs, embed the question, top-k cosine, stuff into a prompt — gets you a demo. It rarely survives contact with real users. The first thing that breaks is recall: your chunks are the wrong size, your query is too short, your embeddings can't tell two similar-sounding policies apart.
The fix is almost always hybrid search (BM25 + dense) followed by a reranker. The reranker is the unlock: a small cross-encoder can move a relevant chunk from rank 47 to rank 2, and that single move turns 'sometimes correct' into 'usually correct'.
After that, invest in the eval harness before you invest in fancy models. A pipeline you can measure beats a pipeline you can tune, every time.