How to self-host a reranker

Production · ~10 min read · Updated 23 Jun 2026

Hosted rerank APIs are convenient, but self-hosting gives you zero per-call cost, full data control, and predictable latency on your own GPU or CPU fleet. Here’s a practical path from a first CrossEncoder call to a production microservice.

When self-hosting wins

• High volume. Reranking 50–100 chunks per query adds up fast on per-1k-doc APIs.
• Data residency. Queries and documents never leave your VPC.
• Custom fine-tuning. You can swap in a domain-tuned checkpoint without waiting on a vendor.

When you need multilingual quality with minimal ops, a hosted API like Cohere Rerank or Jina may still be cheaper all-in — see choose by scenario.

Pick a model

Start with bge-reranker-base on CPU for prototyping. On GPU, A/B Qwen3-Reranker-4B against bge-v2-m3 or mxbai-large on your labelled set — do not assume leaderboard order transfers.

Quick start with sentence-transformers

pip install sentence-transformers torch

from sentence_transformers import CrossEncoder

model = CrossEncoder("BAAI/bge-reranker-base", max_length=512)

query = "How do I add reranking to RAG?"
docs = [
    "Retrieve 50–100 candidates, rerank with a cross-encoder, keep top 5.",
    "London is the capital of the United Kingdom.",
]

pairs = [(query, d) for d in docs]
scores = model.predict(pairs)  # higher = more relevant
ranked = sorted(zip(scores, docs), reverse=True)

Batch your pairs — scoring 50 documents in one predict() call is far faster than 50 separate forwards. Cap passage length at 512 tokens (model default) to avoid silent truncation.

Serving options

# Minimal FastAPI sketch
from fastapi import FastAPI
from sentence_transformers import CrossEncoder

app = FastAPI()
model = CrossEncoder("BAAI/bge-reranker-base")

@app.post("/rerank")
def rerank(body: dict):
    q, docs = body["query"], body["documents"]
    scores = model.predict([(q, d) for d in docs])
    order = sorted(range(len(docs)), key=lambda i: -scores[i])
    return {"results": [{"index": i, "score": float(scores[i])} for i in order]}

For ONNX export and C++ runtimes, see the model cards on Hugging Face. For browser-only experiments, try our in-browser demo with the xsmall ONNX builds.

Ops checklist

• GPU: a single T4 handles ~30–80 ms for 50 docs with bge-base; CPU is 5–10× slower.
• Memory: budget ~1–2 GB VRAM for base models; large variants need more.
• Warm-up: run a dummy predict on startup to avoid cold-start latency spikes.
• Monitoring: track p95 latency and NDCG on a labelled slice — evaluate rerankers.
• Fallback: if the rerank service is down, pass through vector top-k rather than failing the whole RAG request.

Keep reading