How to self-host a reranker
Hosted rerank APIs are convenient, but self-hosting gives you zero per-call cost, full data control, and predictable latency on your own GPU or CPU fleet. Here’s a practical path from a first CrossEncoder call to a production microservice.
When self-hosting wins
- High volume. Reranking 50–100 chunks per query adds up fast on per-1k-doc APIs.
- Data residency. Queries and documents never leave your VPC.
- Custom fine-tuning. You can swap in a domain-tuned checkpoint without waiting on a vendor.
When you need multilingual quality with minimal ops, a hosted API — Cohere, Voyage, or in mainland China Alibaba Cloud Model Studio and SiliconFlow — may still be cheaper all-in. The cost calculator prices your volume; see also choose by scenario.
Pick a model
| Model | Size | Licence | Best for |
|---|---|---|---|
| bge-reranker-v2-m3 | ~568M | Apache 2.0 | Default — multilingual, small enough for CPU |
| gte-multilingual-reranker-base | 306M | Apache 2.0 | Smaller multilingual option (70+ languages, 8K context) |
| Qwen3-Reranker-4B | ~4B | Apache 2.0 | Default GPU pick — in Qwen’s tests, the best of its three sizes on English and instruction-following |
| mxbai-rerank-base-v2 | ~494M | Apache 2.0 | Multilingual incl. Chinese; best size/quality balance of mxbai v2 |
| jina-reranker-v3.5 | ~0.6B | CC BY-NC 4.0 | Listwise, long context — non-commercial unless licensed |
| hs-rerank-1 | 114M | MIT | Compact multilingual CPU option — int8 ONNX, 8,192 tokens, English / Spanish / French / German / Portuguese / Chinese / Japanese / Korean |
Start with bge-reranker-v2-m3, which runs on CPU. On GPU, A/B Qwen3-Reranker-4B against it or mxbai-rerank-base-v2 on your labelled set — do not assume leaderboard order transfers. Until October 2026 this table recommended bge-reranker-base, BAAI’s 2023 model for Chinese and English only.
For a compact multilingual CPU option, consider hs-rerank-1: 114M in total, of which 98M are the embedding table and only 15M the transformer, 8,192 tokens per query-and-passage pair, MIT-licensed, with an int8 ONNX build for CPUs and the deployment config Text Embeddings Inference needs. Hindsight publish its results only as an image on the model card, so there is no score here to compare it by — measure it on your own queries before choosing it over bge-reranker-v2-m3.
A smaller English-only CPU candidate
For English-only search, also test ettin-reranker-17m-v1: 17M, Apache 2.0, with a model-card limit of 7999 tokens per query-and-passage pair. Its publisher reports CPU results using SDPA; this is a self-hosting candidate, not a model added to our browser demo. Keep passages short and measure quality and latency on your own workload.
This is a coverage catch-up: the official release article dates to May 2026. The card reports NanoBEIR and retrieval tests with different first-stage models. Those results do not supply a directly comparable full BEIR average for our table, and English support does not establish Chinese or multilingual quality.
Quick start with sentence-transformers
pip install sentence-transformers torch
from sentence_transformers import CrossEncoder
model = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)
query = "How do I add reranking to RAG?"
docs = [
"Retrieve 50–100 candidates, rerank with a cross-encoder, keep top 5.",
"London is the capital of the United Kingdom.",
]
pairs = [(query, d) for d in docs]
scores = model.predict(pairs) # higher = more relevant
ranked = sorted(zip(scores, docs), reverse=True)
Batch your pairs — scoring 50 documents in one predict() call is far faster than 50 separate forwards. max_length=512 truncates anything longer, so chunk passages to fit rather than letting the end of a passage silently drop off.
Serving options
Embedded in your app
Load the model inside your FastAPI / Flask RAG service. Simplest for low QPS; scale replicas horizontally.
Dedicated rerank microservice
Expose POST /rerank with { query, documents[] }. Share one GPU across many app pods.
# Minimal FastAPI sketch
from fastapi import FastAPI
from sentence_transformers import CrossEncoder
app = FastAPI()
model = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)
@app.post("/rerank")
def rerank(body: dict):
q, docs = body["query"], body["documents"]
scores = model.predict([(q, d) for d in docs])
order = sorted(range(len(docs)), key=lambda i: -scores[i])
return {"results": [{"index": i, "score": float(scores[i])} for i in order]}
You don’t have to write the service yourself: Hugging Face’s Text Embeddings Inference and Infinity both serve cross-encoders behind a rerank endpoint — the gte-multilingual-reranker-base card shows the Docker commands for each. Check that your model is on the server’s supported list first.
For ONNX export and C++ runtimes, see the model cards on Hugging Face. For browser-only experiments, try our in-browser demo with the xsmall ONNX builds.
Ops checklist
- GPU: worth it once you rerank more than a few dozen passages per query. Measure p50 / p95 on your own hardware before sizing — model cards rarely publish latency, and we don’t quote figures we haven’t measured.
- Memory: in fp16 the weights alone take about 2 bytes per parameter — roughly 1.1 GB for bge-reranker-v2-m3’s 568M and 8 GB for a 4B model — plus activations that grow with batch size and passage length.
- Warm-up: run a dummy predict on startup to avoid cold-start latency spikes.
- Monitoring: track p95 latency and NDCG on a labelled slice — evaluate rerankers.
- Fallback: if the rerank service is down, pass through vector top-k rather than failing the whole RAG request.
Prototype before you deploy
Paste your own query and passages in the browser demo — no server required.
Open the demo →