Query Translation: Techniques for Query Expansion and Multi-Query RAG

In building Retrieval-Augmented Generation (RAG) pipelines, we often assume users write perfect queries. In reality, user queries are frequently ambiguous, brief, or poorly structured.

If an LLM relies on a single poorly written query for retrieval, it will fetch irrelevant documents, which directly degrades generation quality. Query Translation addresses this by transforming the user's initial query into one or more refined queries before querying the database.

Query Translation & Expansion Pipeline

Key Query Translation Patterns

Several techniques can translate queries to optimize retrieval:

1. Query Expansion (Multi-Query): Generating multiple alternative formulations of the user's query from different angles to ensure all potential relevant terminology is captured.
2. Sub-Query Decomposition: Breaking a complex multi-part query into simpler sub-queries, retrieving resources for each sub-query individually, and compiling the total context.
3. Query Rewriting (HyDE): Using a hypothetical document embeddings (HyDE) approach to generate a synthetic response and embedding that response instead of the raw query.

The Mathematics of Multi-Query Search

Let the user query be q. A translation function T(q) generates k distinct queries:

T(q) = \{q_1, q_2, \dots, q_k\}

For each q_i, the system retrieves the top N documents:

R(q_i) = \text{Top-N Documents for } q_i

The final set of retrieved documents is the union of all retrieved sets, often filtered or merged using Reciprocal Rank Fusion:

R_{final} = \bigcup_{i=1}^{k} R(q_i)

Python Implementation of Multi-Query Generation

Here is a Python script that takes a user query, generates variations using an LLM, and combines the results:

import openai

def generate_query_variations(query, num_variations=3):
    client = openai.OpenAI()
    
    prompt = (
        "You are an AI assistant helping optimize search engine queries.\n"
        f"Generate {num_variations} alternative versions of the following query.\n"
        "Provide each query on a new line. Do not add numbers or labels.\n"
        f"Original Query: {query}"
    )
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2
    )
    
    variations = response.choices[0].message.content.strip().split("\n")
    return [v.strip() for v in variations if v.strip()]

# Example Usage
original = "How does speculative decoding speed up inference?"
# variations = generate_query_variations(original)
# print("Original:", original)
# print("Variations:", variations)

Implementation Tradeoffs

• Increased Latency: Running an LLM query generation step adds 200ms to 800ms of latency before retrieval begins. This makes it less suitable for real-time search applications unless highly optimized or run asynchronously.
• Cost: Every user query now triggers additional LLM calls and multiple vector retrievals.
• Relevance: Multi-query retrieval can increase retrieval noise if not paired with a strict reranker.

Conclusion

Query translation techniques like query expansion and sub-query decomposition are essential tool additions for handling complex, real-world user queries. By transforming unstructured inputs into targeted searches, they significantly improve the robustness and accuracy of RAG systems.