RAG document reranking

Quick Deployment Strategy

1. First Stage (Recall): Use a fast hybrid retriever (like BM25 + dense embeddings) to pull a broad shortlist of the top 50 to 100 candidate chunks.[ianas.fr](https://redis.io/blog/top-reranking-models-rag-accuracy/)[[2](https://redis.io/blog/top-reranking-models-rag-accuracy)](https://local-ai-zone.github.io/guides/best-ai-reranker-models-ultimate-ranking-2026.html)
2. Second Stage (Precision): Pass that shortlist to your reranker to compress and re-order them down to the top 5 to 10 chunks that actually get fed into your LLM prompt, effectively eliminating "context rot".[ianas.fr](https://redis.io/blog/top-reranking-models-rag-accuracy/)[[2](https://redis.io/blog/top-reranking-models-rag-accuracy)](https://local-ai-zone.github.io/guides/best-ai-reranker-models-ultimate-ranking-2026.html)

To help narrow down the best fit for your stack, tell me:

• Are you looking for a managed API service or a self-hosted open-source model?
• What kind of data are you searching through (e.g., general text, code, medical/legal docs)?
• Which vector database or orchestration framework (LangChain, LlamaIndex, etc.) are you using?

A particularly notable current development is Jina Reranker v3.5: Jina describes it as a 0.6B-parameter multilingual listwise reranker with 131K context, while jina-reranker-v2-base-multilingual remains a smaller cross-encoder option.jina.ai

My practical ranking

1. Voyage Rerank — best default if quality is #1

Voyage's current documentation lists rerank-3 as its highest-accuracy option and rerank-3-lite as the latency/cost-oriented alternative. Both support 32K-token context.docs.voyageai.com

I'd start here if you're building a high-quality RAG system and can use an API.

2. Cohere Rerank — best enterprise-safe choice

Cohere's current lineup has rerank-v4.0-pro for maximum quality and rerank-v4.0-fast for lower latency/high throughput. It also supports multilingual text and semi-structured JSON.docs.voyageai.com+3

It's particularly attractive if your stack already uses Cohere or you want a very straightforward managed RAG component.

3. Jina Reranker — best for long context/self-hosting flexibility

Jina is especially interesting when documents are large or multilingual. Its current v3.5 model has a substantially larger context window than many traditional cross-encoder rerankers.jina.ai

4. BGE — best open-source starting point

If you need to keep data inside your infrastructure, BGE is still an excellent baseline. It's also useful as the comparison point against which I'd benchmark commercial APIs.

5. ColBERT — best when latency/scale dominates

ColBERT's late-interaction architecture is worth considering when reranking tens or hundreds of candidates at very high query volume. It's more of an architecture choice than a simple drop-in API.

How I'd implement it

A good starting pipeline is:

The important point is don't rerank your entire corpus. Rerank only the candidate set produced by your first-stage retriever. Voyage explicitly recommends this pattern: rerank candidates obtained from embedding or lexical retrieval.docs.voyageai.com

Which one I'd choose

• Best overall: Voyage Rerank
• Best enterprise API: Cohere Rerank
• Best long-document option: Jina Reranker
• Best self-hosted: BGE / Jina open-weight models
• Best for extremely high-scale retrieval: ColBERT-style late interaction
• Best low-risk experiment: Benchmark Voyage + Cohere + BGE on your own queries

One caveat: public benchmark claims vary substantially by dataset and vendor methodology. For example, current comparisons disagree on whether Voyage or Cohere leads overall, so I would not choose solely from published leaderboard numbers.andrew.ooo

If you tell me your vector DB (Pinecone, Elasticsearch, Weaviate, pgvector, etc.), expected query volume, and whether you need self-hosting, I can narrow this to the best 2–3 choices and give you a concrete RAG architecture.

The particularly interesting change is Jina Reranker v3/v3.5. Jina reports v3 at 0.6B parameters with 61.94 nDCG@10 on BEIR, ahead of the listed BGE-v2-m3 (56.51) and mxbai-rerank-large-v2 (61.44); the model can process up to 64 documents simultaneously and has a 131K-token context. Its model card explicitly recommends v3.5 for new projects as a stronger drop-in successor.huggingface.co

1. Start with Jina Reranker v3.5

This would be my first experiment for a new RAG system.

It's particularly attractive if you have:

• multilingual documents
• long chunks/documents
• structured documents
• a relatively small candidate set
• a need for strong quality without moving to a multi-billion-parameter model

The important architectural distinction is that Jina v3 isn't simply another traditional BERT-style cross-encoder: it uses causal self-attention across the query/document context and performs listwise reranking, allowing it to consider multiple candidates together.huggingface.co+1

Jina Reranker v3 model card

2. Qwen3-Reranker-4B if quality matters more than latency

If you're willing to spend more GPU compute, Qwen3-Reranker-4B is one of the models I'd put in a serious bake-off.

Jina's published comparison puts it at 61.16 BEIR nDCG@10, while its CoIR score is particularly strong at 73.91. That makes it especially interesting for technical/code-oriented retrieval.huggingface.co+1

It's Apache-2.0 licensed and available for local Transformers inference.huggingface.co

Qwen3-Reranker-4B

I'd use it when retrieval quality is substantially more valuable than ~tens of milliseconds of additional latency.

3. BGE-reranker-v2-m3 for the boring-but-good production baseline

Don't overlook BAAI/bge-reranker-v2-m3.

It's a relatively lightweight multilingual cross-encoder and is explicitly designed for reranking retrieved query/passage pairs. BAAI describes it as easy to deploy with fast inference and strong multilingual performance.huggingface.co

It also has a mature ecosystem through FlagEmbedding, including FP16 inference and fine-tuning support.github.com

BGE Reranker v2 M3

I'd choose BGE when:

• you're self-hosting
• GPU budget is limited
• you need multilingual retrieval
• you want something mature and predictable
• your candidate pool is ~20–100 documents

It's also an excellent baseline against which to measure newer models.

4. Cohere if you don't want to operate the model

If you're happy with a managed API, Cohere Rerank is still a very useful benchmark.

The advantage isn't necessarily that it will always beat every open model. It's that you can get a high-quality reranking stage without managing GPUs, batching, model serving, quantization, etc.

Independent comparisons put Cohere near the top of common BEIR reranking evaluations, although these numbers should be treated as directional rather than universal—the choice of first-stage retriever and dataset changes the ordering.github.com

More important than the model: how you use it

A common mistake is:

I'd generally do:

Reranking cannot recover documents that your first-stage retrieval never retrieved.

So I'd typically retrieve 50–100 candidates, rerank them, and send perhaps 5–15 to the generator. BAAI itself recommends hybrid retrieval followed by reranking, noting that cross-encoder rerankers are more accurate than the underlying bi-encoder retrieval models.github.com

Candidate count matters

For example:

Don't blindly rerank 500–1,000 documents. The whole point is to spend expensive semantic-comparison compute only after cheap retrieval has narrowed the search space.

What I'd actually benchmark

If you're building a serious RAG system, I'd run this four-model bake-off:

1. Jina Reranker v3.5
2. Qwen3-Reranker-4B
3. BGE-reranker-v2-m3
4. Cohere Rerank

And evaluate:

• Recall@20 / Recall@50
• MRR@5
• nDCG@5 / nDCG@10
• answer faithfulness
• answer accuracy
• p50/p95 latency
• tokens/second or requests/sec
• cost/query

Don't select purely on BEIR. Your own queries and documents are much more important.

For example, a reranker that wins BEIR by 2 points but loses on your company's terminology, tables, product IDs, or legal clauses is the wrong model.

One particularly useful diagnostic

Measure:

If Recall@100 is already ~95% but Recall@10 is ~60%, reranking is exactly the right intervention.

If Recall@100 is only 60%, don't spend all your effort swapping rerankers—you have a first-stage retrieval problem. Improve hybrid retrieval, chunking, query rewriting, or embeddings first.

My practical ranking

If I were starting today:

Best overall: Jina Reranker v3.5
Best high-quality self-hosted: Qwen3-Reranker-4B
Best lightweight self-hosted: BGE-reranker-v2-m3
Best managed/API baseline: Cohere Rerank
Best low-latency architecture at huge scale: investigate late-interaction retrieval/reranking (ColBERT-style) rather than conventional cross-encoding.

One caveat: published benchmark numbers are not apples-to-apples across vendors, and even Jina's model card numbers are its own evaluation setup. For example, its table shows BGE v2-m3 doing particularly well on MIRACL while Qwen3-4B does particularly well on CoIR, illustrating why "best reranker" depends heavily on the retrieval domain.huggingface.co+1

If you tell me your corpus type (docs/code/PDFs), language, approximate corpus size, latency target, and whether you can run GPUs, I can narrow this to 2–3 models and give you a concrete RAG retrieval → reranking configuration.

My practical recommendation

If you want the easiest production implementation, start with Cohere rerank-v4.0-pro. Its API is explicitly designed to take a query plus retrieved documents and return them ordered by semantic relevance.docs.cohere.com+1

If your documents are long or you want to benchmark another strong managed option, test Voyage rerank-3. Voyage supports up to 1,000 documents per request and 32K-token query/document limits for its newest rerankers.docs.cohere.com+2

A typical pipeline would be:

The important part is not to rerank your entire corpus. Rerankers jointly process the query and document (cross-encoding), which gives better relevance judgments than ordinary embedding similarity but makes them substantially more expensive than first-stage retrieval.docs.cohere.com+2

For a new system, I'd benchmark Cohere v4 Pro vs. Voyage rerank-3 on your own queries and relevance labels, rather than assuming a generic benchmark winner. Then use the fast/lite variant if the quality difference doesn't justify its latency/cost.

Cohere Rerank API documentation
Voyage Rerank API documentation

If you tell me your retriever (Pinecone/Qdrant/pgvector/Elastic/etc.), candidate count, latency target, and approximate queries/day, I can narrow this to a concrete choice and show the integration code.