“Does my RAG system work?” is the wrong question. The right questions are: does it hallucinate? Does it answer the question? Does it retrieve the right documents? RAGAS gives you a number for each of these — and this lesson shows you how to run it.
RAGAS (Retrieval Augmented Generation Assessment) is an open-source framework that uses LLMs to automatically score RAG pipelines across four orthogonal dimensions. Each metric isolates one failure mode so you know where to improve.
0–1 (higher = better)
“Does the answer only contain information from the retrieved context?”
Measures hallucination. The LLM-judge checks each claim in the answer against the context. A score of 1 means every statement is supported. Score < 0.8 means the model is fabricating.
Answer: "The transformer architecture uses multi-head self-attention."
Context: "Transformers rely on self-attention mechanisms applied in parallel across multiple heads."
Answer: "Transformers were invented in 2017 by Hinton at Google."
Context: "Attention Is All You Need was published in 2017."
# pip install ragas langchain-anthropic
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset
# Build your test dataset (golden set)
test_cases = [
{
"question": "What is the capital of France?",
"contexts": ["Paris is the capital and largest city of France."],
"answer": "The capital of France is Paris.",
"ground_truth": "Paris", # Only needed for context_recall
},
{
"question": "How does gradient descent work?",
"contexts": [
"Gradient descent minimizes a function by iteratively moving in the direction of steepest descent.",
"The learning rate controls the step size during gradient descent updates.",
],
"answer": "Gradient descent works by computing the gradient of the loss and stepping in the negative gradient direction.",
"ground_truth": "Gradient descent minimizes loss by moving in the negative gradient direction using a learning rate.",
},
]
dataset = Dataset.from_list(test_cases)
# Run evaluation
result = evaluate(
dataset=dataset,
metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result.to_pandas()[["faithfulness", "answer_relevancy", "context_precision", "context_recall"]])
# → faithfulness: 0.95 answer_relevancy: 0.91 context_precision: 0.88 context_recall: 0.87| Low Score | Likely Cause & Fix |
|---|---|
| Faithfulness < 0.8 | LLM is hallucinating. Fix: strengthen system prompt ("answer only from context"), add contextual compression, use a less creative model. |
| Answer Relevancy < 0.8 | Answers are indirect or verbose. Fix: improve system prompt, add "be direct" instruction, check if question is ambiguous. |
| Context Precision < 0.7 | Too much irrelevant context retrieved. Fix: tighten chunk size, reduce n_results, add re-ranking, use better hybrid weights. |
| Context Recall < 0.7 | Missing relevant documents. Fix: increase n_results, improve chunking strategy, use multi-query retrieval, check embedding model. |
RAGAS needs ground-truth answers for context recall. Generate them by: (1) picking 20–50 representative questions, (2) manually writing the correct answer for each, (3) storing them as JSON. The investment is worth it — a golden dataset lets you track improvements across every pipeline change.
You have now covered the full advanced RAG toolkit. The capstone project in the next lesson combines hybrid search, re-ranking, contextual compression, and RAGAS evaluation into a production pipeline.