Instruction-following rerankers
Classic cross-encoders answer one question: “how relevant is this passage to this query?” Instruction-following rerankers (e.g. Contextual AI Rerank v2) take an extra control signal — a short instruction that defines what “relevant” means for this task.
Why plain relevance is not enough
Two documents can both “match” a query while serving opposite user goals. “Cancel subscription” might need the self-serve steps for a support bot, or the retention offer for a sales agent. A generic MS MARCO-style reranker was trained on topical relevance — not on task policy.
Instructions let you steer ranking without fine-tuning a new checkpoint for every product surface.
How instruction rerank differs
| Input | Classic cross-encoder | Instruction reranker |
|---|---|---|
| Query | Yes | Yes |
| Passage | Yes | Yes |
| Task instruction | No (or baked into query text) | First-class field |
| Typical deploy | Self-host or API | Mostly hosted API today |
# Conceptual API shape (vendor SDKs differ)
results = client.rerank(
query="cancel my plan",
documents=candidates,
instruction="Prefer official help articles with step-by-step cancel steps; demote marketing and upsell pages.",
) # higher score = better match to query under that instructionYou can approximate this with a classic model by prefixing the query ("[Support KB] cancel my plan"), but dedicated instruction models are trained to treat the control text as constraints rather than bag-of-words topics.
When to use it
- Multi-surface products. Same corpus, different ranking policies (support vs sales vs compliance).
- Hard negatives share vocabulary. Legal “termination for convenience” vs “for cause”; support “refund” vs “exchange”.
- You lack fine-tuning budget. Steering via instructions is cheaper than training domain adapters every quarter.
Skip it when your queries are single-intent FAQ lines and a small cross-encoder (Jina tiny, mxbai xsmall, bge-base) already hits NDCG on your labels — see evaluate rerankers.
Vs classic cross-encoders
- Classic (bge, Cohere, Jina, mxbai): best documented BEIR-style numbers; open weights or mature free tiers; great default for RAG “topical precision”.
- Instruction APIs (Contextual AI, etc.): stronger when relevance is policy-shaped; less portable; harder to compare on public leaderboards — our models table marks BEIR as n/a on purpose.
- Late-interaction (ColBERT): different axis entirely (token MaxSim at scale) — see late-interaction guide.
How to evaluate
Do not paste a generic BEIR number and call it done. Build 30–50 queries with the instruction you will ship, label which docs are correct under that policy, and measure NDCG@5 / MRR before vs after. Swap instruction text alone and check that rankings move in the intended direction.
If changing the instruction does not change the top-5, you are paying for a feature you are not using — fall back to a classic reranker.
See classic cross-encoder behaviour
Our in-browser demo scores topical relevance only (no instruction field). Use it to baseline what a plain reranker does, then A/B an instruction API on your labelled set.
Open the demo →