You retrieve 20 candidates with fast ANN search. Only 3 are genuinely useful. A re-rankerreads every candidate in full context with the query and outputs a precise relevance score — turning a coarse top-20 list into a refined top-3 that the LLM can actually use.
Your embedding model is a bi-encoder: it encodes the query and each document independently, then measures similarity between the resulting vectors. This is fast — you can pre-compute document embeddings offline — but the query never “sees” the document during encoding.
A cross-encoder takes the query and document together as a single input and outputs a relevance score in one forward pass. Because the two texts interact directly through the attention mechanism, the score is far more accurate. The cost: you cannot pre-compute anything — every candidate requires a full forward pass at query time.
| Property | Bi-Encoder | Cross-Encoder |
|---|---|---|
| Offline pre-computation | ✅ (document vectors) | ❌ |
| Query latency for 1M docs | ~5ms (ANN) | ~hours (infeasible) |
| Query latency for top-20 | ~5ms | ~200ms (GPU) |
| Relevance accuracy | Good | Excellent |
| Token interaction | None (separate) | Full cross-attention |
| Best use | First-stage retrieval | Second-stage re-ranking |
Coarse Retrieval (Bi-Encoder)
ANN search over full corpus → top-100 candidates. Fast. Trades some precision for recall.
Re-ranking (Cross-Encoder)
Re-score the top-100 with a cross-encoder → return top-5. Slow but highly precise.
Choose your deployment approach to see the code pattern and trade-offs:
~150ms CPU overhead, ~20ms GPU. Free — runs on your infrastructure.
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def rerank(query: str, candidates: list[str], top_k: int = 5):
pairs = [[query, doc] for doc in candidates]
scores = reranker.predict(pairs)
ranked = sorted(
zip(candidates, scores.tolist()),
key=lambda x: x[1],
reverse=True,
)
return ranked[:top_k]
# Pipeline: retrieve top-20 with ANN → re-rank → return top-5
initial = hybrid_search(query, n_results=20)
candidate_texts = [r["text"] for r in initial]
reranked = rerank(query, candidate_texts, top_k=5)from sentence_transformers import CrossEncoder
# MS-MARCO trained cross-encoder — strong general-purpose re-ranker
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def rerank(query: str, candidates: list[str], top_k: int = 5) -> list[tuple[str, float]]:
"""
Score each candidate against the query and return top_k by score.
candidates: list of raw text strings (not IDs)
returns: list of (text, score) tuples, highest score first
"""
pairs = [[query, doc] for doc in candidates]
scores = reranker.predict(pairs) # Shape: (len(candidates),)
ranked = sorted(
zip(candidates, scores.tolist()),
key=lambda x: x[1],
reverse=True,
)
return ranked[:top_k]
# Usage in a RAG pipeline
initial_results = hybrid_search(query, n_results=20) # coarse retrieval
candidate_texts = [r["text"] for r in initial_results]
reranked = rerank(query, candidate_texts, top_k=4)
for text, score in reranked:
print(f"[{score:.3f}] {text[:80]}")If you prefer not to run a local model, Cohere offers a best-in-class hosted re-ranking API. It accepts text pairs and returns relevance scores — no GPU required on your end.
import cohere # pip install cohere
co = cohere.Client(api_key="YOUR_COHERE_API_KEY")
def cohere_rerank(query: str, candidates: list[str], top_k: int = 5) -> list[dict]:
response = co.rerank(
model="rerank-english-v3.0",
query=query,
documents=candidates,
top_n=top_k,
)
return [
{
"text": candidates[r.index],
"score": r.relevance_score,
"original_rank": r.index,
}
for r in response.results
]When the query touches two themes, initial retrieval returns chunks from each — re-ranking surfaces the most relevant combination.
BM25 inflates scores for long documents that contain the query term many times. Cross-encoders evaluate actual relevance.
"What is the refund policy for premium plans?" may retrieve general policy documents — re-ranking picks the exact clause.
# Typical latency breakdown for a re-ranked RAG pipeline:
#
# ANN search (semantic + BM25 + RRF) → ~15ms
# Re-ranking 20 candidates (local) → ~150ms on CPU, ~20ms on GPU
# Re-ranking 20 candidates (Cohere) → ~300ms network round-trip
# LLM generation → ~1000–3000ms
#
# Total: ~1.5–3.5s end-to-end
#
# Optimization: only re-rank when confidence of top-1 semantic result < 0.75
# This adds a fast early-exit for high-confidence queries
TOP_1_THRESHOLD = 0.75
def smart_rerank(query, initial_results):
if initial_results[0]["distance"] < (1 - TOP_1_THRESHOLD):
return initial_results[:5] # Skip re-ranker — already confident
return rerank(query, [r["text"] for r in initial_results])In the next lesson we tackle a different failure mode: the user's query is ambiguous or uses vocabulary that doesn't match the document. Query transformation — rewriting the query before retrieval — solves this without changing the index.