Query Translation: Techniques for Query Expansion and Multi-Query RAG

In building Retrieval-Augmented Generation (RAG) pipelines, we often assume users write perfect queries. In reality, user queries are frequently ambiguous, brief, or poorly structured.

If an LLM relies on a single poorly written query for retrieval, it will fetch irrelevant documents, which directly degrades generation quality. Query Translation addresses this by transforming the user's initial query into one or more refined queries before querying the database.

Query Translation & Expansion Pipeline

Key Query Translation Patterns

Several techniques can translate queries to optimize retrieval:

1. Query Expansion (Multi-Query): Generating multiple alternative formulations of the user's query from different angles to ensure all potential relevant terminology is captured.
2. Sub-Query Decomposition: Breaking a complex multi-part query into simpler sub-queries, retrieving resources for each sub-query individually, and compiling the total context.
3. Query Rewriting (HyDE): Using a hypothetical document embeddings (HyDE) approach to generate a synthetic response and embedding that response instead of the raw query.

The Mathematics of Multi-Query Search

Let the user query be q. A translation function T(q) generates k distinct queries:

T(q) = \{q_1, q_2, \dots, q_k\}

For each q_i, the system retrieves the top N documents:

R(q_i) = \text{Top-N Documents for } q_i

The final set of retrieved documents is the union of all retrieved sets, often filtered or merged using Reciprocal Rank Fusion:

R_{final} = \bigcup_{i=1}^{k} R(q_i)

Python Implementation of Multi-Query Generation

Here is a Python script that takes a user query, generates variations using an LLM, and combines the results:

import openai

def generate_query_variations(query, num_variations=3):
    client = openai.OpenAI()
    
    prompt = (
        "You are an AI assistant helping optimize search engine queries.\n"
        f"Generate {num_variations} alternative versions of the following query.\n"
        "Provide each query on a new line. Do not add numbers or labels.\n"
        f"Original Query: {query}"
    )
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2
    )
    
    variations = response.choices[0].message.content.strip().split("\n")
    return [v.strip() for v in variations if v.strip()]

def multi_query_retrieve(query, retriever, top_n=5):
    """Run the original query plus its variations, then merge with Reciprocal Rank Fusion."""
    all_queries = [query] + generate_query_variations(query)
    fused_scores = {}

    for q in all_queries:
        for rank, doc_id in enumerate(retriever(q, top_n)):
            # RRF with the standard k=60 smoothing constant
            fused_scores[doc_id] = fused_scores.get(doc_id, 0) + 1 / (60 + rank + 1)

    return sorted(fused_scores, key=fused_scores.get, reverse=True)

# Example Usage
if __name__ == "__main__":
    original = "How does speculative decoding speed up inference?"

    variations = generate_query_variations(original)
    print("Original:", original)
    for i, v in enumerate(variations, 1):
        print(f"  Variation {i}: {v}")

    # A stub retriever standing in for your vector database client
    corpus = {
        "doc_a": "Speculative decoding uses a small draft model to propose tokens.",
        "doc_b": "KV caching avoids recomputing attention keys and values.",
        "doc_c": "Draft model acceptance rate determines the realized speedup.",
    }

    def retriever(q, top_n):
        terms = set(q.lower().split())
        ranked = sorted(
            corpus,
            key=lambda d: len(terms & set(corpus[d].lower().split())),
            reverse=True,
        )
        return ranked[:top_n]

    print("Fused ranking:", multi_query_retrieve(original, retriever))

Implementation Tradeoffs

• Increased Latency: Running an LLM query generation step adds 200ms to 800ms of latency before retrieval begins. This makes it less suitable for real-time search applications unless highly optimized or run asynchronously.
• Cost: Every user query now triggers additional LLM calls and multiple vector retrievals.
• Relevance: Multi-query retrieval can increase retrieval noise if not paired with a strict reranker.

When Not to Reach for This

Query translation is frequently the second thing you should try, not the first. If your retrieval is underperforming, the usual culprit is chunking — chunks that are too large dilute the embedding, chunks that split mid-argument strand the answer across two neighbours. Multi-query expansion papers over that by casting a wider net, which means you pay an LLM call on every request to compensate for an indexing problem you could have fixed once, offline. Measure recall@50 on a fixed eval set before and after a chunking change; if that number moves, fix the index first.

The failure mode people underestimate is expansion drift. Ask for four rephrasings of a narrow question and the model will generalise — "how do I rotate a Postgres replication slot's credentials" becomes "what is Postgres replication," and now the union contains three documents about replication basics that outrank the one specific page you needed. Union-then-rerank is not optional here; without a cross-encoder pruning the merged set back down, expansion reliably makes precision worse even as it improves recall. HyDE is the sharpest instance of this tradeoff: it works beautifully on domains the model knows and generates plausible, confidently wrong hypothetical documents on your internal jargon, which then retrieve confidently wrong neighbours.

A practical rule: only expand when the retrieved set is thin or low-scoring. Run the plain query first, check whether the top result clears a similarity floor, and skip the whole translation layer when it does. Most queries in a production system are easy, and the cheapest expansion is the one you never ran.