How to self-host a reranker

Production · ~10 min read ·

Hosted rerank APIs are convenient, but self-hosting gives you zero per-call cost, full data control, and predictable latency on your own GPU or CPU fleet. Here’s a practical path from a first CrossEncoder call to a production microservice.

When self-hosting wins

When you need multilingual quality with minimal ops, a hosted API — Cohere, Voyage, or in mainland China Alibaba Cloud Model Studio and SiliconFlow — may still be cheaper all-in. The cost calculator prices your volume; see also choose by scenario.

Pick a model

ModelSizeLicenceBest for
bge-reranker-v2-m3~568MApache 2.0Default — multilingual, small enough for CPU
gte-multilingual-reranker-base306MApache 2.0Smaller multilingual option (70+ languages, 8K context)
Qwen3-Reranker-4B~4BApache 2.0Default GPU pick — in Qwen’s tests, the best of its three sizes on English and instruction-following
mxbai-rerank-base-v2~494MApache 2.0Multilingual incl. Chinese; best size/quality balance of mxbai v2
jina-reranker-v3.5~0.6BCC BY-NC 4.0Listwise, long context — non-commercial unless licensed
hs-rerank-1114MMITCompact multilingual CPU option — int8 ONNX, 8,192 tokens, English / Spanish / French / German / Portuguese / Chinese / Japanese / Korean

Start with bge-reranker-v2-m3, which runs on CPU. On GPU, A/B Qwen3-Reranker-4B against it or mxbai-rerank-base-v2 on your labelled set — do not assume leaderboard order transfers. Until October 2026 this table recommended bge-reranker-base, BAAI’s 2023 model for Chinese and English only.

For a compact multilingual CPU option, consider hs-rerank-1: 114M in total, of which 98M are the embedding table and only 15M the transformer, 8,192 tokens per query-and-passage pair, MIT-licensed, with an int8 ONNX build for CPUs and the deployment config Text Embeddings Inference needs. Hindsight publish its results only as an image on the model card, so there is no score here to compare it by — measure it on your own queries before choosing it over bge-reranker-v2-m3.

A smaller English-only CPU candidate

For English-only search, also test ettin-reranker-17m-v1: 17M, Apache 2.0, with a model-card limit of 7999 tokens per query-and-passage pair. Its publisher reports CPU results using SDPA; this is a self-hosting candidate, not a model added to our browser demo. Keep passages short and measure quality and latency on your own workload.

This is a coverage catch-up: the official release article dates to May 2026. The card reports NanoBEIR and retrieval tests with different first-stage models. Those results do not supply a directly comparable full BEIR average for our table, and English support does not establish Chinese or multilingual quality.

Quick start with sentence-transformers

pip install sentence-transformers torch

from sentence_transformers import CrossEncoder

model = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

query = "How do I add reranking to RAG?"
docs = [
    "Retrieve 50–100 candidates, rerank with a cross-encoder, keep top 5.",
    "London is the capital of the United Kingdom.",
]

pairs = [(query, d) for d in docs]
scores = model.predict(pairs)  # higher = more relevant
ranked = sorted(zip(scores, docs), reverse=True)

Batch your pairs — scoring 50 documents in one predict() call is far faster than 50 separate forwards. max_length=512 truncates anything longer, so chunk passages to fit rather than letting the end of a passage silently drop off.

Serving options

Embedded in your app

Load the model inside your FastAPI / Flask RAG service. Simplest for low QPS; scale replicas horizontally.

Dedicated rerank microservice

Expose POST /rerank with { query, documents[] }. Share one GPU across many app pods.

# Minimal FastAPI sketch
from fastapi import FastAPI
from sentence_transformers import CrossEncoder

app = FastAPI()
model = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

@app.post("/rerank")
def rerank(body: dict):
    q, docs = body["query"], body["documents"]
    scores = model.predict([(q, d) for d in docs])
    order = sorted(range(len(docs)), key=lambda i: -scores[i])
    return {"results": [{"index": i, "score": float(scores[i])} for i in order]}

You don’t have to write the service yourself: Hugging Face’s Text Embeddings Inference and Infinity both serve cross-encoders behind a rerank endpoint — the gte-multilingual-reranker-base card shows the Docker commands for each. Check that your model is on the server’s supported list first.

For ONNX export and C++ runtimes, see the model cards on Hugging Face. For browser-only experiments, try our in-browser demo with the xsmall ONNX builds.

Ops checklist

Prototype before you deploy

Paste your own query and passages in the browser demo — no server required.

Open the demo →

Keep reading