How to evaluate rerankers
Public BEIR scores are a starting point, not a verdict. A reranker that wins on English news may flop on your support tickets. You need a small labelled set of real queries and two metrics: NDCG@k (ranking quality) and MRR (is the best answer near the top?).
Build a labelled query set
Pull 30–100 real queries from logs (anonymised). For each query, mark which document IDs are relevant (binary is enough to start). Store as JSON:
{
"q-001": {
"query": "How do I reset my API key?",
"relevant": ["doc-42", "doc-108"]
}
}Thirty queries with one gold document each is enough to see whether reranking beats retrieval-only — you’re measuring relative lift, not publishing a leaderboard.
Metrics that matter
| Metric | What it tells you | Typical k |
|---|---|---|
| NDCG@k | Rewards putting highly relevant docs at the top; graded relevance if you have it | 5 or 10 |
| MRR | How high the first relevant doc ranks — great for single-answer RAG | — |
| Recall@k | Is the right doc in the shortlist at all? Diagnoses retrieval, not reranker | 50–100 |
Run metrics before and after reranking on the same retrieved candidates. If NDCG@5 jumps but Recall@50 is low, your problem is retrieval — reranking can’t invent missing chunks.
Evaluation protocol
- Fix a retrieval pipeline (vector-only or hybrid) and log top-50 candidates per query.
- Score baseline: order as retrieved (or bi-encoder scores).
- Apply reranker A, measure NDCG@5 and MRR.
- Swap reranker B (different model or hosted API), repeat.
- Track latency p50/p95 alongside quality — a 2-point NDCG gain may not justify 400 ms.
Python with ranx
from ranx import Qrels, Run, evaluate
qrels = Qrels.from_file("qrels.json") # ground truth
run_before = Run.from_file("retrieve.json")
run_after = Run.from_file("rerank.json")
for name, run in [("retrieve", run_before), ("rerank", run_after)]:
print(name, evaluate(qrels, run, ["ndcg@5", "mrr@10"]))ir-measures and BEIR’s evaluator work too. The important part is consistent qrels and the same candidate pool — not which library you pick.
Comparing rerankers
Use the model comparison table for ballpark BEIR numbers, then validate on your labelled set. Hosted APIs (Cohere, Jina, Voyage) are fastest to A/B; open models (bge, mxbai) need GPU for fair latency comparison.
Regression tests: check eval metrics into CI. When you change chunk size, embedding model, or reranker version, re-run the same JSON qrels — quality drops should block deploys.
Build intuition before you benchmark
Use the live demo to see how a cross-encoder reorders a hand-picked shortlist — then scale up with labelled queries.
Try the demo →