You retrieve a 500-word chunk. Only 30 words actually answer the question. The other 470 words consume LLM context and dilute the signal. Contextual compression uses an LLM to extract only the relevant sentences from each retrieved chunk before passing them to the generator.
Every token in your context window costs money and competes for the LLM's attention. When a retrieved chunk contains mostly background information surrounding the one relevant sentence, you are paying for noise and giving the LLM more text to misinterpret.
Compressing chunks before generation can:
Select a compression approach to see the code and cost trade-offs:
Best quality. Adds 100–300ms + haiku cost per chunk. Saves 60–80% on generator tokens.
import anthropic
client = anthropic.Anthropic()
PROMPT = """Extract ONLY the sentences that directly answer the question.
If no sentences answer it, respond exactly: NOT RELEVANT
Question: {question}
Chunk: {chunk}
Extracted:"""
def compress_chunk(question: str, chunk: str) -> str | None:
response = client.messages.create(
model="claude-haiku-4-5-20251001", # Fast/cheap for compression
max_tokens=500,
messages=[{"role": "user", "content": PROMPT.format(
question=question, chunk=chunk)}],
)
result = response.content[0].text.strip()
return None if result == "NOT RELEVANT" else result
# 8 chunks × 500 tokens → ~1,200 compressed tokens (70% savings)
# Generation cost: 1,200 × $0.003/1K = $0.0036 vs $0.012 uncompressedimport anthropic
client = anthropic.Anthropic()
COMPRESSION_PROMPT = """Given the following document chunk and a question, extract ONLY the sentences
from the chunk that directly answer the question. Do not add any new information.
If no sentences in the chunk answer the question, respond with exactly: "NOT RELEVANT"
Question: {question}
Chunk:
{chunk}
Extracted relevant sentences:"""
def compress_chunk(question: str, chunk: str) -> str | None:
"""
Extract the relevant portion of a chunk for the given question.
Returns None if the chunk is not relevant.
"""
response = client.messages.create(
model="claude-haiku-4-5-20251001", # Use fast/cheap model for compression
max_tokens=500,
messages=[{
"role": "user",
"content": COMPRESSION_PROMPT.format(question=question, chunk=chunk),
}],
)
result = response.content[0].text.strip()
return None if result == "NOT RELEVANT" else result
def compressed_retrieve(question: str, initial_results: list[dict]) -> list[str]:
"""Compress a list of retrieved chunks and filter out non-relevant ones."""
compressed = []
for result in initial_results:
relevant_text = compress_chunk(question, result["text"])
if relevant_text:
compressed.append(relevant_text)
return compressed
# Full pipeline:
# 1. Retrieve 8 candidates with hybrid search
candidates = hybrid_search(question, n_results=8)
# 2. Compress each to only the relevant sentences
compressed_chunks = compressed_retrieve(question, candidates)
# 3. Feed compressed chunks to LLM — much shorter context!
context = "\n\n---\n\n".join(compressed_chunks)
print(f"Original token estimate: ~{sum(len(r['text']) for r in candidates) // 4}")
print(f"Compressed token estimate: ~{len(context) // 4}")from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor
from langchain_anthropic import ChatAnthropic
from langchain_community.vectorstores import Chroma
llm = ChatAnthropic(model="claude-haiku-4-5-20251001")
compressor = LLMChainExtractor.from_llm(llm)
base_retriever = Chroma(...).as_retriever(search_kwargs={"k": 8})
compression_retriever = ContextualCompressionRetriever(
base_compressor=compressor,
base_retriever=base_retriever,
)
# Automatically retrieves 8 docs and compresses to only relevant sentences
docs = compression_retriever.get_relevant_documents(
"What is the return policy for electronics?"
)
for doc in docs:
print(doc.page_content) # Already compressed!When LLM compression is too slow or expensive, use keyword-based sentence filtering as a fast baseline:
import re
from sentence_transformers import CrossEncoder
# Use a cross-encoder to score individual sentences
sentence_reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def sentence_compress(question: str, chunk: str, threshold: float = 0.3) -> str:
"""Extract sentences above relevance threshold."""
sentences = [s.strip() for s in re.split(r"[.!?]+", chunk) if len(s.strip()) > 20]
if not sentences:
return chunk
pairs = [[question, s] for s in sentences]
scores = sentence_reranker.predict(pairs)
relevant = [s for s, score in zip(sentences, scores) if score > threshold]
return " ".join(relevant) if relevant else chunkScenario: 8 chunks × 500 tokens each = 4,000 input tokens per query At $0.003/1K input tokens (claude-sonnet-4-6): Without compression: 4,000 tokens = $0.012/query After 70% compression: 1,200 input tokens per query With compression (using haiku at $0.00025/1K): $0.0003 compression cost Generation cost: 1,200 tokens = $0.0036 Total: $0.0039/query — 67% savings Break-even: compression always wins if your corpus chunks are > ~200 tokens and you retrieve > 4 chunks per query.
In Lesson 7, we stop building the pipeline and start measuring it. You will build a RAGAS evaluation suite that quantifies faithfulness, answer relevancy, context precision, and context recall across your entire RAG system.