Context Compression: Reducing RAG Costs and Improving Context Relevance

Large Language Models (LLMs) continue to expand their context windows, with some supporting up to 1 million tokens or more. However, just because you can feed hundreds of pages of context into an LLM does not mean you should.

Adding large amounts of raw text to a prompt introduces major drawbacks:

1. Financial Cost: API billing is directly tied to input token counts.
2. Latency: Reading large context windows increases Time-to-First-Token (TTFT) and processing times.
3. Information Loss: LLMs often suffer from the "lost in the middle" effect, failing to notice crucial details buried deep inside long prompts.

Context compression addresses this by filtering out irrelevant details and redundant language before sending context to the LLM.

Context Compression & Filtering

The Concepts behind Prompt Compression

The objective of prompt compression is to reduce a source text T to a shorter sequence T_c such that the semantic information content is preserved:

I(T_c) \approx I(T) \quad \text{and} \quad |T_c| \ll |T|

This is achieved using two main methodologies:

• Heuristic Filtering: Sentence-level or word-level filtering using a similarity metric or ranker (e.g. Cross-Encoder) to drop low-scoring sentences.
• Information-Theoretic Compression: Using a small, lightweight language model (like Llama-3-8B) to compute the perplexity of each token in the prompt. Tokens with low perplexity (e.g. "the", "and", redundant nouns) represent predictable context and can be pruned with minimal impact on LLM comprehension.

Implementing Context Compression with LLMLingua

Microsoft's llmlingua library is the leading framework for perplexity-based prompt compression. Here is an implementation:

from llmlingua import PromptCompressor

# Initialize compressor with a small model
# compressor = PromptCompressor(
#     model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meeting",
#     use_auth_token=False
# )

original_prompt = (
    "The system architecture consists of a front-end interface built on React, "
    "communicating over HTTPS with a back-end RESTful API constructed in Python. "
    "The back-end server connects to a PostgreSQL database for persistent storage, "
    "and uses Redis for session caching and rate-limiting. This setup ensures high availability "
    "and low latency. Additionally, a backup server replicates the main database every hour."
)

# Compress prompt
# result = compressor.compress_prompt(
#     context=[original_prompt],
#     instruction="Summarize the system stack.",
#     target_token=60,
#     conserve_ratio=0.4
# )
# print("Compressed Prompt:", result["compressed_prompt"])

Tuning Compression Configurations

To configure context compression in production, consider:

1. Target Ratio: A compression ratio of 2x to 3x (reducing token count by 50% to 66%) usually preserves accuracy while significantly cutting costs.
2. Model Selection: Using small models like llmlingua-2 (which uses token classification models based on BERT) provides fast, low-latency compression compared to generative causal models.
3. Task Specificity: Prompts containing code or structured JSON require lower compression ratios than conversational prose to prevent breaking syntax.

Conclusion

Context compression is an essential tool for production RAG systems. Pruning redundant tokens using information-theoretic and heuristic compression reduces API costs, cuts latency, and improves accuracy by reducing noise in LLM context windows.