Evaluating RAG Pipelines: How to Measure Retrieval and Generation Quality

Deploying a Retrieval-Augmented Generation (RAG) pipeline is straightforward. Optimizing it is not. Because RAG combines two distinct AI paradigms—unstructured search (retrieval) and generative text modeling (generation)—traditional software test suites are insufficient.

A user might receive a bad answer because the search engine retrieved the wrong documentation (a retrieval failure), or because the LLM misread the correct documentation and made up a fact (a generation failure).

To debug and optimize these systems, engineers use structured evaluation frameworks like Ragas and TruLens. These frameworks break evaluation down into measurable components known as the RAG Triad. This article defines these critical metrics and demonstrates how to implement an automated "LLM-as-a-Judge" evaluation script.

---

The RAG Triad of Evaluation

The RAG Triad evaluates the interactions between three primary entities: the User Query, the Retrieved Context, and the Generated Answer.

Evaluating RAG Pipelines: RAG evaluation pipeline showing context recall, faithfulness, and answer relevance checks

1. Context Relevance (Precision & Recall)

Measures the quality of the retrieval stage.

• Context Precision: Out of all the retrieved chunks, how many were actually relevant? High precision means minimal noise is sent to the LLM.
• Context Recall: Did the system retrieve all the necessary facts required to answer the question? If the source database contains five steps to configure a system, but retrieval only fetches two, context recall is low.

2. Faithfulness (Groundedness)

Measures the generation stage.

• Definition: Is the generated answer derived solely from the retrieved context? Or did the LLM hallucinate outside information?
• Calculation: The generated answer is split into individual statements. Each statement is analyzed to determine if it is directly supported by the retrieved context.

3. Answer Relevance

Measures the alignment between the question and the final response.

• Definition: Does the generated answer actually address the user's query, or does it dance around the topic? This is measured by comparing the semantic similarity of the generated response to the intent of the original question.

---

LLM-as-a-Judge: The Mechanics of Automated Scoring

Evaluating thousands of production queries manually using human labelers is slow and expensive. Modern frameworks use an LLM-as-a-Judge pattern. By prompting a powerful model (like GPT-4) with strict guidelines, we can extract quantitative scores for qualitative metrics.

For example, to calculate Faithfulness:

1. Ask the judge LLM to split the generated answer into a list of independent declarative sentences.
2. For each sentence, ask the judge to output a binary decision: Yes (supported by retrieved context) or No (not supported by context).
3. Compute the score:

\text{Faithfulness} = \frac{\text{Number of supported statements}}{\text{Total number of statements}}

---

Python Implementation: Building an Evaluator

The following Python script implements a complete, runnable evaluation pipeline. It splits a generated answer into individual claims and uses a structured mock checker (which mimics an LLM evaluation prompt) to score the Faithfulness and Context Relevance of the response.

import json
from typing import List, Dict, Any

class RAGEvaluator:
    def __init__(self):
        # In production, these methods would make API calls to GPT-4/Claude 
        # using strict prompt templates and structured JSON schema parsing.
        pass

    def evaluate_faithfulness(self, context: str, answer: str) -> Dict[str, Any]:
        """
        Measures if the answer contains any information not present in the context.
        """
        # Step 1: Split answer into distinct statements (simulating LLM extraction)
        statements = [
            "Python 3.10 is required for deployment.",
            "NodeJS is required for the frontend application.",
            "You must run the install script to start."
        ]
        
        # Step 2: Grade each statement against the context
        # (Mocking LLM classification output based on context matching)
        grades = []
        for statement in statements:
            # Let's check if the statement can be found in our mock context
            if "Python 3.10" in statement and "Python 3.10" in context:
                grades.append({"statement": statement, "supported": True})
            elif "install script" in statement and "install script" in context:
                grades.append({"statement": statement, "supported": True})
            else:
                # NodeJS is not mentioned in our retrieved context, representing a hallucination
                grades.append({"statement": statement, "supported": False})
                
        supported_count = sum(1 for g in grades if g["supported"])
        score = supported_count / len(statements)
        
        return {
            "metric": "Faithfulness (Groundedness)",
            "score": score,
            "statements_evaluated": grades
        }

    def evaluate_context_relevance(self, query: str, retrieved_contexts: List[str]) -> Dict[str, Any]:
        """
        Measures what fraction of the retrieved chunks are directly useful to the query.
        """
        relevance_grades = []
        for idx, chunk in enumerate(retrieved_contexts):
            # Check if keywords from query exist in chunk
            is_relevant = any(word in chunk.lower() for word in query.lower().split())
            relevance_grades.append({
                "chunk_index": idx,
                "relevant": is_relevant
            })
            
        relevant_count = sum(1 for g in relevance_grades if g["relevant"])
        score = relevant_count / len(retrieved_contexts)
        
        return {
            "metric": "Context Precision (Relevance)",
            "score": score,
            "details": relevance_grades
        }

# Execution
if __name__ == "__main__":
    # Test Data
    user_query = "What python version is needed to deploy the backend?"
    
    retrieved_docs = [
        "System requirements: Python 3.10 or higher must be installed to run the backend libraries.",
        "Frontend assets are hosted in AWS S3 buckets."  # Irrelevant chunk
    ]
    
    generated_response = (
        "To deploy the backend, Python 3.10 is required. "
        "Additionally, NodeJS is required for the frontend. "
        "Make sure to execute the install script."
    )
    
    evaluator = RAGEvaluator()
    
    # 1. Evaluate context relevance (Stage 1)
    context_results = evaluator.evaluate_context_relevance(user_query, retrieved_docs)
    print("\n--- Context Relevance Evaluation ---")
    print(json.dumps(context_results, indent=2))
    
    # 2. Evaluate generation faithfulness (Stage 2)
    joined_context = " ".join(retrieved_docs)
    faithfulness_results = evaluator.evaluate_faithfulness(joined_context, generated_response)
    print("\n--- Faithfulness Evaluation ---")
    print(json.dumps(faithfulness_results, indent=2))

---

Establishing Continuous Evaluation Loops

To scale evaluation across your engineering cycles:

1. Curate an Evaluation Dataset: Assemble a "Golden Dataset" of 100-200 representative query-context-ground truth triplets.
2. Track Metrics via CI/CD: Every time a prompt is edited or database chunking parameters are changed, run your evaluator against the golden dataset. Ensure new configurations do not regress faithfulness or context recall scores before merging to production.

---

What These Numbers Don't Tell You

It is worth being blunt about the limits of LLM-as-a-judge, because teams tend to trust these scores more than they have earned.

Judge models are noisy. Score the same answer twice and you often get different numbers; run the same evaluation next week and the provider may have updated the model underneath you. That makes small movements meaningless. A faithfulness score going from 0.82 to 0.85 after a prompt change is almost certainly noise, and treating it as a win is how teams end up shipping regressions. Run each evaluation several times, look at the spread, and decide up front how large a change has to be before you act on it. If you cannot distinguish your improvement from the variance, you do not have an improvement yet.

They also inherit their judge's biases. Models rate their own family's output higher, prefer longer and more confidently worded answers, and are influenced by position when comparing options. If you use GPT-4 to judge a GPT-4 pipeline, the scores lean generous in a direction you cannot see from the numbers alone.

The deeper limitation is that faithfulness only measures whether the answer matches the retrieved context. An answer that is perfectly grounded in a document your team deprecated eighteen months ago scores 1.0. So does a technically-supported answer that omits the safety caveat sitting in a chunk retrieval never returned. Groundedness is not correctness, and it is certainly not usefulness.

The practical discipline is this: use automated metrics as a regression alarm, not as a definition of quality. They are good at telling you something changed. Read fifty real production traces by hand every month — not sampled from your golden set, sampled from actual users — and you will find failure categories no metric on this page was designed to catch. Then add those to the golden set. The evaluation suite should be a record of mistakes you have already made, growing over time, rather than a fixed scoreboard you optimize against.