What is a Reranker?

Why the right passage comes back eighth

A reranker is a second scoring pass over results a search has already returned. It reads your question and each candidate document together, one pair at a time, and reorders them by how well that document answers the question you actually asked. Without one, the passage that answers your question can rank below the top few your pipeline keeps, and the model never sees it.

🧭 Part 13 of the 🔍 RAG & Search course

TL;DR

• Your index decided what each document was about before your question existed. It matched your question against that decision, and nothing since has revisited it.

• So the passage you needed comes back eighth. In a search box you would scroll down to it. A RAG pipeline keeps the top five and throws the rest away, so the eighth result may as well not exist.

• The reranker judges after the question arrives. It reads your question and one passage side by side, so it can see whether that passage answers this question, and it repeats that for every candidate on the shortlist.

• You can add it this afternoon. Every other way to improve retrieval sends you back over the whole corpus. A reranker slots in between the search and the prompt and touches nothing else.

• Feed it too many candidates and it loses to the search it was fixing. Past a certain shortlist size it starts burying the answer instead of finding it, and the size is yours to find.

You Have Already Done This By Hand

Somebody asks whether your retention policy covers deleted accounts, so you search the company docs. Eight results come back. The policy overview. The deletion FAQ. The 2023 version of the policy. You skim past the privacy policy, past storage limits, past a summary of the European privacy rules, past a guide to closing an account. The eighth one is the clause you needed: deleted accounts are the exception, and the policy does not cover them. You click it and get on with your day.

You have reranked search results a thousand times. You just did it with your eyes, in the seconds it took to read eight titles.

A RAG pipeline cannot do that. It takes the top 5 and puts them in the prompt. Result eight does not exist. There is no scrolling, no skimming, and no second look. Whatever the ranking put in the top five is everything the model has to answer from.

None of the first five mention the exception, so the model answers from those. It tells your user that deleted accounts are covered, which is the opposite of what the policy says. You log it as a hallucination and go rewrite the prompt. The model never saw the clause.

Why Your Index Got It Wrong

An embedding model reads a piece of text and writes out a list of numbers that places its meaning. Engineers call that list a vector, and texts about the same thing land close together. Your index runs every document through the model at ingest and keeps its vector. When you type a query, it runs the query through the same model and returns whichever documents landed nearest. The two sides never meet inside the model, and that shape has a name: the bi-encoder. We compared the databases that hold those vectors in Pinecone vs Weaviate vs Qdrant vs Milvus.

Search is fast because the model did the hard work at ingest, long before you asked anything. To do that work early, it had to squeeze everything the passage might ever be relevant to into a few hundred numbers, without knowing which of those things you would ask about.For your retention policy, the model recorded one fact: this document is about data retention. A filing decision cannot say whether the fourth paragraph carves out deleted accounts, because nobody had asked yet.

So the index ranks by topic. You asked a question. Every passage that is broadly about the right subject outranks the one paragraph that answers you, and the score gives you no way to tell them apart.

⚠️ Confusion Alert: A bigger embedding model does not fix this, so the upgrade you were about to price out buys you nothing here. No embedding model ever sees your query and the passage together. A better one just files more precisely, and it still files before you ask.

What the Second Pass Reads

A reranker glues your query and one document together and runs the pair through a model as a single input. Now the model sees your question sitting next to the passage, so it can check whether this clause answers that question. Engineers call a model that reads a pair this way a cross-encoder.

You pay for reading the pair together in work you cannot bank. A bi-encoder embeds a document once and reuses that vector forever. A cross-encoder scores one query against one document, then throws the score away the moment the next query arrives. Ten million documents would cost ten million runs through the model on every single search. Nobody reranks a corpus. You rerank a shortlist, and everything that follows comes down to how big a shortlist you can afford.

Feeding both halves in as one input means they share one length limit. The published usage examples for bge-reranker-v2-m3 cut that pair off at 512 tokens, and plenty of teams ship the default untouched. Feed it a 900-token chunk and it scores the first 500 and never reads the last 400, so the clause you needed is judged on a paragraph that does not contain it. Nothing errors. The reranker ranks it low, you see worse retrieval than before you added it, and you go tuning the model that was never the problem.

Self-hosted, the model never sees the tokens past its limit. Hosted, it sees them and charges you for every extra 500. Cohere counts every 500 tokens as another document, so 1,000-token chunks cost double what you budgeted for. Check your chunk size before you check anything else¹.

So you chain them. Your query hits the index, and vector search, keyword search, or the two fused together (hybrid search) returns 100 to 150 candidates. The cross-encoder scores each of them against your query, you keep the top 5 to 20, and those go in the prompt. Perplexity runs that exact chain at production scale, which we took apart in How Perplexity Built Their Search Engine.

Against a hosted reranker, the scoring step is one call:

import cohere
co = cohere.ClientV2()

reranked = co.rerank(
    model="rerank-v4.0-fast",
    query="does the retention policy cover deleted accounts?",
    documents=candidates,   # the 100 chunks your vector search just returned
    top_n=5,
)

That call is the whole integration. Choosing what sits behind it takes less thought than the catalog suggests:

• No GPUs of your own? Call Cohere’s rerank-v4.0-fast and stop thinking about it. rerank-v4.0-pro is there for the day you have measured that ordering quality is your bottleneck².

• Already running GPUs? Self-host bge-reranker-v2-m3. It is 0.6 billion parameters under an Apache 2.0 license, and it sits beside your embedding model without adding a vendor³.

• Already buying from Voyage? rerank-2.5 is as good as either one. Picking it is a procurement question rather than an engineering one⁴.

Whichever you pick, your pipeline looks the same. The open question is how much it buys you.

How Much It Actually Buys You

Almost nobody publishes the reranker’s isolated contribution, because it lands last and gets folded into whatever else shipped that quarter. Anthropic separated it. They counted a run as a failure when the needed chunk never showed up in the top 20 the model gets, then walked that number down one change at a time⁵:

• Plain embeddings: 5.7% of questions missed

• Prepend a line to each chunk naming its source document: 3.7%

• Run keyword search alongside the vectors: 2.9%

• Retrieve 150 chunks, rerank, keep the top 20: 1.9%

That last step cut a third of the remaining failures, and it is the only one you could ship today. Every gain above it meant another pass over the corpus. This one is a call you insert between the search and the prompt, and then you measure.

What Can Go Wrong (and What’s Overhyped)

Feeding it more candidates stops helping, then starts hurting. One team tested four rerankers against three retrievers across academic and enterprise datasets, scaling how many documents each one scored. In 53% of the academic experiments and 44% of the enterprise ones, the reranker ended up worse than the plain retriever at the largest candidate count⁶. Rerankers are trained on short lists where nearly every passage is plausible. Hand one a thousand candidates and most of them are junk, and sooner or later one junk passage scores high by accident and pushes the real answer out of the top 20. Run your eval at 50, 100 and 200 candidates and keep whichever wins. The peak usually sits in there.

It cannot retrieve what retrieval missed. The reranker only ever sees the shortlist your search handed it, so if the passage that answers the question came back at rank 300 and you fetched 150, no reranker recovers it. This is the failure that looks like the reranker underperforming: you add it, your eval barely moves, and you go sweeping candidate counts when the answer was never in the candidate set. Measure recall at your fetch depth before you touch anything else. If the right passage is not in the 150, the problem is your search or your chunker, and reordering cannot fix either.

It works well enough that you stop looking. Reranking buys more retrieval quality per afternoon than anything else left in RAG, and that is exactly why teams stop looking once they have added it. You add the call, the eval ticks up, retrieval gets marked solved, and the chunker that cuts tables in half never gets opened again. That is how a team loses a quarter to a bug it had already found. Add the reranker this afternoon. Then go open the chunker you have been avoiding.

The One Thing to Remember

A reranker does not find better documents. The documents were already there, sitting in the candidate list your search returned. A reranker rereads them with your question in hand, which is the one thing your index could never do, because your index made up its mind about every passage before your question existed.

So whoever fetched the candidates set your ceiling. The reranker only ever reorders what the search already handed it.

💬 What is your candidate count, and did you ever sweep it? Reply or comment. I want to know how many teams are running the default 100 without checking.

Where to Next?

• What are Embeddings? The prerequisite. What that fixed list of numbers actually encodes, and why comparing two of them means anything.

• What is Semantic Search? The stage that hands the reranker its candidates, and why hybrid search beats vectors alone.

• What is Chunking? The ceiling above every ranker. What the splitter throws away, no reordering recovers.

🔜 Next Tuesday: What Breaks When You Self-Host an LLM. The production gaps that show up after the deploy tutorial ends.

FAQ

How much latency does reranking add?
One run through the model per candidate, plus a network round trip when you call a hosted reranker. Two things drive the number: how many candidates you send, and how long each one is, because the model reads every token of the pair. So 50 candidates cost about half of 100, and doubling the length of each candidate roughly doubles the work again. Measure your own before you promise anyone a number.

What is the difference between a reranker and an embedding model?
An embedding model encodes your query and your documents separately, so your pipeline computes document vectors at ingest and reuses them for every query. A reranker reads a query and a document together in one pass and scores that specific pair. The embedding model is fast enough to search millions of passages and blind to your exact question. The reranker sees the question and costs one run through the model per document, so it only runs on a shortlist.

Do I need a reranker if I already use hybrid search?
Usually yes, and they stack. Hybrid search improves which candidates come back. The reranker improves the order they come back in. In Anthropic’s numbers, hybrid search took the failure rate from 3.7% to 2.9%, and reranking on top of it took it to 1.9%.

Is ColBERT a reranker?
It is a different road to the same job. ColBERT embeds every token separately at ingest, then scores a pair by matching those per-token vectors against each other, so it skips running the pair through a model at all. Engineers call that late interaction. That is a real speed win and a real tuning project. Reach for it when reranking latency is blocking your release, never as your first reranker.

Can I use an LLM as a reranker instead?
You can, by asking a general model to score or order the passages. It is flexible and it is the expensive option, since you pay full generation pricing for a ranking a purpose-built cross-encoder produces for a fraction of the cost. Reach for it when you need scoring rules a trained reranker has no way to know about.

1. Best practices for using Rerank, Cohere docs

2. Cohere’s Rerank v4.0 Model is Here!, Cohere changelog (December 2025)

3. rerank-2.5 and rerank-2.5-lite: instruction-following rerankers, Voyage AI (Februray 2024)

4. BAAI/bge-reranker-v2-m3, Hugging Face (August 2025)

5. Introducing Contextual Retrieval, Anthropic (September 2024)

6. Drowning in Documents: Consequences of Scaling Reranker Inference, arXiv (November 2024)