Reranking in RAG: What It Is, How It Works, and How to Tune It
Written By:
Matt Tanner
Published on
Your vector search found the right chunk. It just ranked it seventh, and you only send the top three to the model. That's one of the less obvious ways a retrieval-augmented generation (RAG) pipeline fails: the answer is in your knowledge base, retrieval even touched it, and the LLM never saw it.
Reranking adds another scoring pass over retrieved candidates. This guide covers reranker mechanisms, hosted and self-hosted options, and how retrieval depth (k), retained results (n), and model choice interact, including results from a published 40-config sweep.
What Is Reranking in RAG?
Reranking in RAG is a second pass over retrieved documents. After a fast first-stage retrieval returns the top k candidates, a reranking model scores each one for relevance to the user query, and only the top n highest-scoring chunks go into the LLM's context window.
That's two-stage retrieval: a cheap semantic search narrows millions of chunks to a few dozen, then an expensive scorer picks the best handful. The idea predates RAG. Nogueira and Cho's 2019 BERT passage re-ranker took the top spot on the MS MARCO passage leaderboard. In RAG, though, the reranker isn't ordering a results page for someone who can scroll. It decides which few chunks the model reads at all.
Say a user asks whether lost baggage is covered, and up to what limit. Vector search returns 20 chunks about baggage, delays, and passports, and a reranker moves the one stating the actual limit to the top so it survives the cut to three.
Why RAG Needs a Reranker
Retrieval has two jobs that pull in opposite directions. Recall wants a wide net, since a chunk you never retrieve can't help. Precision wants a short list, because every extra chunk costs tokens and adds noise the model has to read past.
Vector search does much of its work before the user asks a question. A bi-encoder converts each chunk into a vector at indexing time. At query time, it embeds the question and searches for nearby chunk vectors.
This is efficient, but similarity is not the same as answering the question. A passage about delayed baggage may be close to a question about lost baggage without stating the relevant coverage limit. The first-stage model therefore matters too: choosing an embedding model for RAG deserves as much care as choosing the reranker.
A cross-encoder reranker scores the query and candidate together, allowing it to consider interactions that independent embeddings miss. It requires a model pass for every pair, so measure its latency at the candidate count and chunk lengths you expect.
Sending every retrieved chunk to the model is not always a good solution. It increases the prompt size and leaves the model to find the useful evidence among irrelevant passages.
Position can matter as well. In the Lost in the Middle study, the tested models often used information at the beginning or end of a long context more effectively than information in the middle, including models designed for long contexts. This is a reason to evaluate context selection and ordering for your model. Reranking lets you retrieve a wider pool, keep a smaller set, and put its highest-scoring evidence first.
How Reranking Works: Retrieve, Rerank, Generate
The reranking step slots between retrieval and prompt assembly in a standard RAG pipeline:
Retrieve the top k candidates (say 20) with a similarity search against your vector database.
Pair the user query with each candidate chunk.
Score each pair with the reranking model.
Keep the top n most relevant chunks (say 3), sorted by score.
Generate the answer from those n chunks.
Here are steps 2 to 4 on their own, using sentence-transformers 6.1.0 and a small MS MARCO cross-encoder. rank() pairs the query with every document and returns them sorted by relevance score.
Run the pip command in a Python 3.12+ virtual environment.
Then save and run the example as a Python script.
It downloads the model on first use, performs inference locally without an API key, and prints the top two candidates.
Scores can vary slightly across package versions and hardware.
In this call, top_k=2 is the number of results to keep: the n used throughout this guide. The baggage-cover passage ranks first, but the second result is a passport rule that does not answer the question. Keeping two results is not automatically better than keeping one.
The displayed scores are raw model outputs, called logits, rather than probabilities. The relevant passage scores 8.33; the other four candidates score between -7.70 and -9.06. Compare these scores within this query. A cutoff such as 0.5 would not mean a 50% probability of relevance.
Types of Rerankers
How a reranker scores candidates affects its accuracy, latency, and hosting requirements.
Type | How it scores | Accuracy | Latency and cost | Good for |
|---|---|---|---|---|
Cross-encoder | Reads query and chunk together, one score per pair | High | One model pass per candidate | A strong default second stage |
Late interaction (ColBERT-style) | Separate multi-vector encodings, matched cheaply | Close to cross-encoders | Precomputed document vectors; bigger index | Large candidate pools, tight latency |
LLM reranker | An LLM grades, compares, or orders candidates | Can match or beat supervised rerankers | High for large prompted LLMs; varies by model and method | Low-volume queries where quality beats latency |
Hosted reranking API | Vendor-hosted model behind an endpoint | Depends on the model | Vendor pricing plus a network hop | Teams that don't want to run inference |
MMR (diversity reranking) | Trades relevance against diversity | No stronger relevance signal | Cheap, runs inside vector search | Removing near-duplicate chunks |
Cross-Encoders
A cross-encoder reranker feeds the query and one candidate through a transformer as a single input and outputs a relevance score instead of an embedding. The score depends on both texts, so a new query means scoring each candidate again. You can't precompute a document vector and search it later, as you can with a bi-encoder. On the BEIR zero-shot benchmark, re-ranking and late-interaction models achieved the best average results, but required more computation.
Late-Interaction Models
ColBERT (Khattab and Zaharia, 2020) encodes the query and document independently, then runs a cheap interaction step over their fine-grained representations. Because document representations are computed offline, it gets close to cross-encoder quality at a fraction of the query-time compute, at the cost of storing multiple vectors per document.
LLM-Based Rerankers
An LLM can judge candidates in several ways: grade each passage separately (pointwise), choose between two passages (pairwise), or order a list of passages (listwise).
RankGPT demonstrates the listwise approach. It gives the model numbered passages and asks for an order such as [2] > [3] > [1], rather than a relevance score. For longer lists, it repeats this over sliding windows of candidates. Using GPT-4 this way added substantial setup latency. Compact models trained specifically for ranking, such as Qwen3-Reranker and the listwise jina-reranker-v3.5, provide another option to evaluate.
Reranking APIs vs. Open-Source Models
The hosting decision is separate from the scoring method. Hosted options include Cohere's rerank-v4.0-pro and rerank-v4.0-fast and Voyage's rerank-3 and rerank-3-lite. Voyage's rerankers are cross-encoders: an API can provide the same joint query-document scoring described above.
With an API, the vendor runs inference and receives your chunks. Self-hosting keeps inference in your environment, but you need serving capacity and a suitable license. For example, Jina's v3.5 weights use CC BY-NC 4.0, so commercial self-hosting requires a separate agreement.
Best Reranker Models for RAG
No single reranker is best; it depends on your languages, chunk lengths, latency budget, and license. These reranking models are worth comparing first:
cross-encoder/ms-marco-MiniLM-L6-v2is the small English baseline, trained on MS MARCO passage ranking and Apache-2.0 licensed. Our FiQA and SciFact tutorials use it.BAAI/bge-reranker-v2-m3is a lightweight, multilingual, Apache-2.0 bge reranker.Qwen/Qwen3-Reranker-0.6B(also 4B and 8B) covers 100+ languages, handles 32k tokens of context, and accepts task instructions.mixedbread-ai/mxbai-rerank-base-v2and-large-v2are Apache-2.0, multilingual, long-context models that load into sentence-transformers'CrossEncoder.jinaai/jina-reranker-v3.5is a 0.6B listwise model that scores many candidates in one forward pass, with non-commercial weights.Cohere rerank-v4.0 and Voyage rerank-3 are two hosted options.
Start with the small baseline plus one multilingual or long-context model, and let your eval set decide, because leaderboard rank is a weak predictor of what wins on your data.
How to Tune Reranking: Retrieval Depth (k), Kept Results (n), and Latency
With a reranker in place, you have three knobs (the model, k, and n) that you can't tune alone because they sit in a chain. Chunk size changes what the embedder sees, retrieval depth changes what the reranker works with, and the reranker changes what the language model finally reads. Even when retrieval finds every relevant chunk, selecting the final context determines which evidence the generator can use.
For a travel-insurance chatbot built by a global leader in enterprise data and analytics, we kept the generator and embedder fixed and compared 40 retrieval configurations in the experiment linked in the introduction.
Nearly all the strongest were retrieve-then-rerank designs. The winner split the policy into 512-token chunks, retrieved the top 10 by similarity, and let a cross-encoder keep the best 2. That lifted the retrieval composite from 0.439 to 0.627, about 43 percent. The benchmark was 76 question-answer pairs synthesized from the policy, making this a controlled demonstration whose winning chunk size and k are specific to that policy.
Pick k and n Together
k sets your recall ceiling, because the reranker can only promote a chunk that retrieval returned. n sets how much reaches the model, and with it your prompt size and how much noise rides along. If k is smaller than n, there's nothing to cut, and if n equals k, the reranker only reorders.
Published starting points vary: 10 down to 2 in the case study, 15 down to 2 or 5 in our FiQA tutorial, 10 down to 5 in our SciFact tutorial, and 20 down to a top_n of 3 in LangChain's reranker docs. Treat them as a grid to test.
In RapidFire, the reranker model, k, and n are ordinary knobs in the LangChain RAG spec, and each can take a list of values.
This is a configuration fragment for a RapidFire RAG tutorial, not a standalone script:
First, complete the RAG installation and initialization steps.
Define
loader,splitter, andembedding_cfgfor your corpus.
The fragment only defines the search space; the tutorial's generator configuration, metric functions, and run_evals() call execute it.
The CPU setting below avoids requiring a CUDA GPU; use cuda:0 only with a compatible GPU and PyTorch installation.
That's 3 × 2 × 3 = 18 combinations. RapidFire automatically drops the ones where k is less than top_n. Here, that's the two with k = 5 and top_n = 8, so the other 16 run on the same eval set.
Test the Reranker Against Your Chunking and Embedding Choices
A cross-encoder truncates input past its max length. For ms-marco-MiniLM-L6-v2, that's 512 tokens for the query and chunk combined, so with large chunks it scores only the opening of each one. Keep chunks inside the reranker's window or test a long-context reranker. Our guide to semantic chunking for RAG covers the splitting side.
A model's benchmark ranking does not tell you which one will work best on your documents. In the case study's second application, a global automotive manufacturer's factory-floor assistant, we compared 432 configurations across seven settings. The winning configuration used cross-encoder/ms-marco-TinyBERT-L6 to keep 5 of the 10 candidates returned by score-threshold search.
TinyBERT was not the strongest model on the reference benchmark. It scored 69.57 on TREC DL 2019's NDCG@10 measure, compared with 74.30 for MiniLM-L6-v2. Those models were trained on MS MARCO, which uses Bing search queries rather than factory documentation. The practical lesson is to test the model together with your retrieval settings on your own data.
Measure the Gain with Retrieval Metrics
Check whether reranking improved the evidence before judging the generated answer. Recall at n measures how much relevant evidence survived into the kept results. Mean reciprocal rank (MRR) tracks how early the first relevant result appears. Normalized discounted cumulative gain (NDCG) rewards putting highly relevant results near the top.
Our FiQA tutorial computes NDCG, MRR, precision, and recall in Python metric functions you can reuse. For the definitions and calculations, see our guide to RAG evaluation metrics.
Keep a no-reranker baseline so you can see what the extra stage changed. In RapidFire, get_context(..., use_reranker=False) returns context without reranking for that comparison. Check both coverage and ordering: in the case study's ablation, removing the reranker let more grounding context through, but ranking precision and the composite score fell.
Choose a configuration on your development set, then confirm the result on held-out questions that did not guide the choice. If the generator samples at nonzero temperature, repeat the answer evaluations to account for variation. The winning configuration is best on the data you measured, not necessarily on every future query.
Watch Latency and Cost
Start with a latency budget, then compare quality within it. A cross-encoder scores every candidate, so increasing k adds reranking work. Increasing n mainly adds tokens to the generation prompt. Model size also affects the tradeoff.
The published MS MARCO model comparison provides a useful example. On its TREC DL 2019 benchmark, the 6-layer MiniLM model scored 74.30 NDCG@10 and processed about 1,800 documents per second on a V100. The 12-layer version scored 74.31 and processed about 960 documents per second on the same hardware. That is a very small quality difference for roughly half the throughput in that test. Measure with your own hardware and chunk lengths before choosing the larger model.
You can also inspect an evaluation before every configuration finishes. For RAG evaluations, RapidFire provides running metric estimates with confidence intervals for supported metric types and ranges. Results update as groups of examples, called shards, finish. Shuffle a grouped or sorted dataset first to reduce bias in the early estimates.
Use that evidence to stop weak configurations or Clone-Modify a promising one with a different reranker_cfg or k. These comparisons happen offline, before deployment.
When Reranking Is Not Worth It
Prove you need a reranker before adding one in these cases:
Your corpus is small, k is already tiny, and retrieval metrics are near their ceiling without one.
A strict latency SLA leaves no room for an extra model pass per candidate, especially with an LLM reranker.
The right chunk isn't in the top k at all. A reranker can't fix recall, so fix chunking or the embedding model, or add keyword search, first.
You'd call a hosted API at high query volume, where the vendor bill outweighs a small quality gain.
Conclusion
Retrieve wide, rerank narrow, and measure. Compare the reranker model, k, and n together on your own data, with the same chunks, embedder, and latency budget.
To try that loop, pip install rapidfireai and run the FiQA tutorial, which compares two reranker top-n values on a financial Q&A dataset and also runs as a no-install Colab notebook. If you're planning this for a production RAG pipeline, talk to our team.
FAQ
What Is Reranking in RAG?
Reranking in RAG is a second-stage filter. After vector search returns the top k retrieved documents, a reranking model, usually a cross-encoder, scores each one against the user query, and only the top n reach the LLM's context window. You retrieve wide for recall, then rerank narrow for precision.
What Are the Best Reranker Models for RAG?
Start with cross-encoder/ms-marco-MiniLM-L6-v2 for English, BAAI/bge-reranker-v2-m3 for multilingual data, Qwen3-Reranker or mxbai-rerank-v2 for longer context, and Cohere or Voyage for a hosted API. The best one for your system is whichever wins on your own eval set within your latency budget.
Can LLMs Be Used as Rerankers?
Yes. You can prompt an LLM pointwise, pairwise, or listwise, and methods like RankGPT have the model output an ordered list of passages directly. LLM rerankers can match or beat supervised rerankers on benchmarks, but prompting a large general-purpose LLM adds substantial latency and cost, so compact models trained for ranking are often more practical.
What Is the Difference Between a Cross-Encoder and a Bi-Encoder?
A bi-encoder embeds the query and each document separately, so you can precompute document vectors and search with cosine similarity. A cross-encoder reads the query and document together and outputs one relevance score, which requires a model pass for every pair at query time. That's why bi-encoders retrieve, and cross-encoders rerank.
Is Reranking the Same as Hybrid Search?
No. Hybrid search runs keyword and vector retrieval and fuses their ranked lists, for example with reciprocal rank fusion. Reranking happens after retrieval and rescores whatever came back, so the two stack: hybrid search can improve recall, then a reranker sorts the combined pool. See our guide to hybrid search for RAG.