Advanced~10 hours8 lessons

LLM Evaluation & Testing

Learn how to know if your AI system actually works. Build eval suites, use LLM-as-judge, run prompt regression tests, and set up production evaluation loops.

What you'll build: an automated LLM assertion and validation test suite

Start Course

Curriculum

Module A: Evaluation Fundamentals

3 lessons

Lesson 1: Why LLM Evaluation Is Hard

Understand why traditional software testing breaks for LLMs — stochastic outputs, no ground truth, and subjective quality. Learn the evaluation taxonomy: unit, integration, and human evals.

50 min

Lesson 2: Benchmarks, Metrics & Evaluation Frameworks

Survey BLEU, ROUGE, exact match, MMLU, and task-specific metrics. Understand what public benchmarks measure vs. what matters for your application.

70 min

Lesson 3: The LLM-as-Judge Pattern

Use a powerful LLM to score another model's outputs on a rubric. Build pairwise comparison evals and absolute scoring evals. Understand calibration and position bias pitfalls.

75 min

Recommended Watch

How to Evaluate LLMs

Build evaluation datasets and use LLM-as-judge patterns to measure model quality systematically.

DeepLearning.AI12:14

Module B: Testing AI Systems

3 lessons

Lesson 4: Unit Testing LLM Applications

Write pytest-based tests for LLM functions using deterministic assertions, regex checks, and LLM-graded assertions. Build a test suite that runs in CI without burning tokens.

75 min

Lesson 5: Prompt Regression Testing

Snapshot-test your prompts so refactoring never silently breaks behavior. Design a golden dataset, store expected outputs, and diff new outputs against baselines.

60 min

Lesson 6: Adversarial Testing & Red-Teaming

Stress-test your AI system with edge cases, jailbreak attempts, prompt injection, and out-of-distribution inputs. Build an adversarial test battery into your eval suite.

60 min

Recommended Watch

Testing AI Systems in Practice

Practical strategies for testing AI systems — from unit tests to adversarial red-teaming.

Arize AI15:42

Module C: Production Evals

2 lessons

Lesson 7: Running Evals in Production

Sample live traffic, log inputs and outputs, run async evals, and set up alerts when quality degrades. Understand the online eval loop that keeps production AI systems healthy.

65 min

Lesson 8: Capstone — Build a Complete Eval Suite

Design and implement a full evaluation suite for a RAG-based Q&A system: golden dataset, unit tests, LLM-as-judge scorer, regression snapshot, and a CI workflow that runs on every PR.

120 min

Recommended Watch

LLM Evaluation at Scale

Set up automated evaluation pipelines and detect prompt regressions before they reach production.

Weights & Biases18:35

Course Quiz

5 questions · ~5 min · Pass 4/5 to unlock your badge

Start Quiz →