Context Compression: Reducing RAG Costs and Improving Context Relevance

Large Language Models (LLMs) continue to expand their context windows, with some supporting up to 1 million tokens or more. However, just because you can feed hundreds of pages of context into an LLM does not mean you should.

Adding large amounts of raw text to a prompt introduces major drawbacks:

1. Financial Cost: API billing is directly tied to input token counts.
2. Latency: Reading large context windows increases Time-to-First-Token (TTFT) and processing times.
3. Information Loss: LLMs often suffer from the "lost in the middle" effect, failing to notice crucial details buried deep inside long prompts.

Context compression addresses this by filtering out irrelevant details and redundant language before sending context to the LLM.

Context Compression & Filtering

The Concepts behind Prompt Compression

The objective of prompt compression is to reduce a source text T to a shorter sequence T_c such that the semantic information content is preserved:

I(T_c) \approx I(T) \quad \text{and} \quad |T_c| \ll |T|

This is achieved using two main methodologies:

• Heuristic Filtering: Sentence-level or word-level filtering using a similarity metric or ranker (e.g. Cross-Encoder) to drop low-scoring sentences.
• Information-Theoretic Compression: Using a small, lightweight language model (like Llama-3-8B) to compute the perplexity of each token in the prompt. Tokens with low perplexity (e.g. "the", "and", redundant nouns) represent predictable context and can be pruned with minimal impact on LLM comprehension.

Implementing Context Compression with LLMLingua

Microsoft's llmlingua library is the leading framework for perplexity-based prompt compression. Here is an implementation:

from llmlingua import PromptCompressor

# Initialize compressor with a small token-classification model
compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meeting",
    use_llmlingua2=True
)

original_prompt = (
    "The system architecture consists of a front-end interface built on React, "
    "communicating over HTTPS with a back-end RESTful API constructed in Python. "
    "The back-end server connects to a PostgreSQL database for persistent storage, "
    "and uses Redis for session caching and rate-limiting. This setup ensures high availability "
    "and low latency. Additionally, a backup server replicates the main database every hour."
)

# Compress the prompt down to a target budget
result = compressor.compress_prompt(
    context=[original_prompt],
    instruction="Summarize the system stack.",
    question="",
    target_token=60
)

print("Compressed Prompt:", result["compressed_prompt"])
print("Original tokens:", result["origin_tokens"])
print("Compressed tokens:", result["compressed_tokens"])
print("Compression ratio:", result["ratio"])

Tuning Compression Configurations

To configure context compression in production, consider:

1. Target Ratio: A compression ratio of 2x to 3x (reducing token count by 50% to 66%) usually preserves accuracy while significantly cutting costs.
2. Model Selection: Using small models like llmlingua-2 (which uses token classification models based on BERT) provides fast, low-latency compression compared to generative causal models.
3. Task Specificity: Prompts containing code or structured JSON require lower compression ratios than conversational prose to prevent breaking syntax.

Conclusion

Run the numbers before you build this. Compression is not free: you are adding a BERT-sized model to the request path, which means either GPU capacity you were not previously renting or 50-150ms of CPU inference per call. If your prompts average 2,000 tokens, compressing to 800 saves a fraction of a cent per request. You need serious volume before that clears the cost of the hardware and the operational surface of another model in production.

Compression also breaks on some content in ways that are easy to miss in testing. Perplexity-based pruning drops predictable tokens, and structured text is full of them: JSON punctuation, code keywords, SQL syntax, table delimiters. A compressor that removes a closing brace or a column header produces context the LLM will still cheerfully reason over, just incorrectly. The same problem applies to legal, medical, and financial text, where "not" and "except" and "prior to" are exactly the low-information-looking function words that get pruned first and exactly the words that invert the meaning. Restrict compression to prose, and hold structured payloads out of it entirely.

There is usually a cheaper fix available first. If your prompts are bloated because you retrieve 20 chunks and send them all, retrieve fewer and re-rank — that reduces tokens without a second model and improves relevance at the same time. If they are bloated because conversation history accumulates unbounded, truncate or summarize the history on a schedule. Prompt compression is the right tool when you have already tightened retrieval and still face genuinely long, genuinely necessary prose context. Reaching for it before that is optimizing the wrong layer.