RAG reranking: when a second pass pays off
In retrieval-augmented generation (RAG), a question is matched against passages from your documents, and the best passages are handed to a language model to answer from. Most of the quality of the answer is decided before the model writes a word: did the right passage make it into the context?
The usual first step ranks passages by embedding similarity. It is fast, and it is coarse. Reranking adds a second, slower and more precise pass over a short list.
Why a second pass can help
An embedding model encodes the question and each passage separately into vectors, then compares the vectors. That is cheap, so you can search thousands of passages. But the model never sees the question and the passage together, so it can rank a passage that shares topic and vocabulary above one that actually answers the question.
A cross-encoder reads the question and one passage together and scores how well that passage answers it. It is much more precise, but you can only afford to run it on a handful of candidates. The pattern is: retrieve wide, rerank narrow.
How our demo does it
The reranking demo runs entirely in your browser:
- Your text is split into passages of about 800 characters with a small overlap.
- A small embedding model scores every passage against your question, and the top 8 become candidates.
- A cross-encoder reranker scores each of the 8 (question, passage) pairs directly. The first run downloads about 15 MB.
- The top 3 passages after reranking each generate their own answer with a small instruct model, so you can compare which passage actually answered the question.
The plain retrieval-augmented Q&A demo skips steps 3 and 4 and answers from the top 4 passages by similarity alone. Putting the two side by side on the same document is the quickest way to see what the second pass changes.
When it tends to help, and when it doesn’t
It tends to help when the right passage is usually somewhere in your top 5 to 10 results but not first, your documents contain many passages on the same topic, and a wrong answer costs more than a slower one.
It will not help when the right passage is not in the candidate pool at all. A reranker only reorders what retrieval found, so if the answer is being missed, fix chunking, overlap or the embedding model first. It also adds latency and another model to run.
What we do and don’t claim
We built the demo to show the mechanism, not to benchmark it, and we do not publish accuracy figures for it. We can say this much from our own project work: in our HCC coding pipeline, a bigger reranker made results worse and was dropped, along with a zero-shot verifier. That is a different task from question answering, so it proves nothing about yours. It is a reminder that reranking is a hypothesis to test, not a default.
How we would decide
- Start with plain retrieval and look at real questions.
- Tune chunk size and overlap next. Our chunk overlap demo lets you see the effect directly.
- Collect a small set of real questions, a few dozen, where you know which passage contains the answer.
- Measure how often the right passage is in the top results with and without the reranker, and what it costs in latency.
- Keep the reranker only if the gain is worth the extra model and the wait.
Try it
Open the reranking demo, paste a document, ask a question, and compare the three reranked answers. Then try the plain Q&A demo on the same text.