Evaluating RAG Pipelines: How to Measure Retrieval and Generation Quality

Deploying a Retrieval-Augmented Generation (RAG) pipeline is straightforward. Optimizing it is not. Because RAG combines two distinct AI paradigms—unstructured search (retrieval) and generative text modeling (generation)—traditional software test suites are insufficient.

A user might receive a bad answer because the search engine retrieved the wrong documentation (a retrieval failure), or because the LLM misread the correct documentation and made up a fact (a generation failure).

To debug and optimize these systems, engineers use structured evaluation frameworks like Ragas and TruLens. These frameworks break evaluation down into measurable components known as the RAG Triad. This article defines these critical metrics and demonstrates how to implement an automated "LLM-as-a-Judge" evaluation script.

---

The RAG Triad of Evaluation

The RAG Triad evaluates the interactions between three primary entities: the User Query, the Retrieved Context, and the Generated Answer.

Evaluating RAG Pipelines: RAG evaluation pipeline showing context recall, faithfulness, and answer relevance checks

1. Context Relevance (Precision & Recall)

Measures the quality of the retrieval stage.

• Context Precision: Out of all the retrieved chunks, how many were actually relevant? High precision means minimal noise is sent to the LLM.
• Context Recall: Did the system retrieve all the necessary facts required to answer the question? If the source database contains five steps to configure a system, but retrieval only fetches two, context recall is low.

2. Faithfulness (Groundedness)

Measures the generation stage.

• Definition: Is the generated answer derived solely from the retrieved context? Or did the LLM hallucinate outside information?
• Calculation: The generated answer is split into individual statements. Each statement is analyzed to determine if it is directly supported by the retrieved context.

3. Answer Relevance

Measures the alignment between the question and the final response.

• Definition: Does the generated answer actually address the user's query, or does it dance around the topic? This is measured by comparing the semantic similarity of the generated response to the intent of the original question.

---

LLM-as-a-Judge: The Mechanics of Automated Scoring

Evaluating thousands of production queries manually using human labelers is slow and expensive. Modern frameworks use an LLM-as-a-Judge pattern. By prompting a powerful model (like GPT-4) with strict guidelines, we can extract quantitative scores for qualitative metrics.

For example, to calculate Faithfulness:

1. Ask the judge LLM to split the generated answer into a list of independent declarative sentences.
2. For each sentence, ask the judge to output a binary decision: Yes (supported by retrieved context) or No (not supported by context).
3. Compute the score:

\text{Faithfulness} = \frac{\text{Number of supported statements}}{\text{Total number of statements}}

---

Python Implementation: Building an Evaluator

The following Python script implements a complete, runnable evaluation pipeline. It splits a generated answer into individual claims and uses a structured mock checker (which mimics an LLM evaluation prompt) to score the Faithfulness and Context Relevance of the response.

import json
from typing import List, Dict, Any

class RAGEvaluator:
    def __init__(self):
        # In production, these methods would make API calls to GPT-4/Claude 
        # using strict prompt templates and structured JSON schema parsing.
        pass

    def evaluate_faithfulness(self, context: str, answer: str) -> Dict[str, Any]:
        """
        Measures if the answer contains any information not present in the context.
        """
        # Step 1: Split answer into distinct statements (simulating LLM extraction)
        statements = [
            "Python 3.10 is required for deployment.",
            "NodeJS is required for the frontend application.",
            "You must run the install script to start."
        ]
        
        # Step 2: Grade each statement against the context
        # (Mocking LLM classification output based on context matching)
        grades = []
        for statement in statements:
            # Let's check if the statement can be found in our mock context
            if "Python 3.10" in statement and "Python 3.10" in context:
                grades.append({"statement": statement, "supported": True})
            elif "install script" in statement and "install script" in context:
                grades.append({"statement": statement, "supported": True})
            else:
                # NodeJS is not mentioned in our retrieved context, representing a hallucination
                grades.append({"statement": statement, "supported": False})
                
        supported_count = sum(1 for g in grades if g["supported"])
        score = supported_count / len(statements)
        
        return {
            "metric": "Faithfulness (Groundedness)",
            "score": score,
            "statements_evaluated": grades
        }

    def evaluate_context_relevance(self, query: str, retrieved_contexts: List[str]) -> Dict[str, Any]:
        """
        Measures what fraction of the retrieved chunks are directly useful to the query.
        """
        relevance_grades = []
        for idx, chunk in enumerate(retrieved_contexts):
            # Check if keywords from query exist in chunk
            is_relevant = any(word in chunk.lower() for word in query.lower().split())
            relevance_grades.append({
                "chunk_index": idx,
                "relevant": is_relevant
            })
            
        relevant_count = sum(1 for g in relevance_grades if g["relevant"])
        score = relevant_count / len(retrieved_contexts)
        
        return {
            "metric": "Context Precision (Relevance)",
            "score": score,
            "details": relevance_grades
        }

# Execution
if __name__ == "__main__":
    # Test Data
    user_query = "What python version is needed to deploy the backend?"
    
    retrieved_docs = [
        "System requirements: Python 3.10 or higher must be installed to run the backend libraries.",
        "Frontend assets are hosted in AWS S3 buckets."  # Irrelevant chunk
    ]
    
    generated_response = (
        "To deploy the backend, Python 3.10 is required. "
        "Additionally, NodeJS is required for the frontend. "
        "Make sure to execute the install script."
    )
    
    evaluator = RAGEvaluator()
    
    # 1. Evaluate context relevance (Stage 1)
    context_results = evaluator.evaluate_context_relevance(user_query, retrieved_docs)
    print("\n--- Context Relevance Evaluation ---")
    print(json.dumps(context_results, indent=2))
    
    # 2. Evaluate generation faithfulness (Stage 2)
    joined_context = " ".join(retrieved_docs)
    faithfulness_results = evaluator.evaluate_faithfulness(joined_context, generated_response)
    print("\n--- Faithfulness Evaluation ---")
    print(json.dumps(faithfulness_results, indent=2))

---

Establishing Continuous Evaluation Loops

To scale evaluation across your engineering cycles:

1. Curate an Evaluation Dataset: Assemble a "Golden Dataset" of 100-200 representative query-context-ground truth triplets.
2. Track Metrics via CI/CD: Every time a prompt is edited or database chunking parameters are changed, run your evaluator against the golden dataset. Ensure new configurations do not regress faithfulness or context recall scores before merging to production.