LLM Evaluation & Testing • Module A: Evaluation FundamentalsLesson 1: Why LLM Evaluation Is Hard
You can write a test for a sorting function. You cannot write a test for “writes good emails.” LLM evaluation requires a completely different mindset — and most engineers underestimate how hard it is until production breaks in ways their test suite never caught.
Select a property to explore why it makes LLM evaluation hard and how to handle it.
The same input produces different outputs on every call. Even at temperature=0, different sampling implementations, prompt caching, and hardware can produce different outputs. You cannot snapshot-test raw model output the way you snapshot-test deterministic functions.
Example
Bad: assert response == "Paris" — this will flake. Good: assert "Paris" in response.lower() or use an LLM judge to verify the answer contains the correct city.
# BAD: snapshot test — will fail intermittently
def test_capital_bad():
resp = llm("What is the capital of France?")
assert resp == "The capital of France is Paris." # Exact match fails
# GOOD: semantic assertion — robust to phrasing variation
def test_capital_good():
resp = llm("What is the capital of France?")
# Option 1: substring check for deterministic facts
assert "paris" in resp.lower()
# Option 2: run multiple times and check consistency
responses = [llm("What is the capital of France?") for _ in range(5)]
assert all("paris" in r.lower() for r in responses), "Inconsistent factual recall"LLM evaluation exists on a spectrum from cheap-and-imprecise to expensive-and-accurate. A mature eval suite uses all three tiers strategically.
| Tier | Method | Cost | Use For |
|---|---|---|---|
| 1 — Unit | Exact match, regex, JSON schema validation | Near zero | Deterministic assertions (format, structure, required fields) |
| 2 — Automated | LLM-as-judge, RAGAS, BLEU/ROUGE | Low–Medium | Quality scoring at CI scale — runs on every PR |
| 3 — Human | Expert annotation, user feedback | High | Calibrating automated evals, catching subtle failures |
Goodhart's Law in AI: once a measure becomes a target, it ceases to be a good measure. Common traps in LLM evaluation:
BLEU rewards n-gram overlap. A model that generates exact reference sentences but adds hallucinated sentences still scores well.
Users click helpful for confident-sounding answers regardless of accuracy. Confident hallucinations score highly.
If your eval set does not include out-of-distribution questions, the model memorizes the distribution and fails in production.
# A mature LLM eval suite has these properties:
# 1. Multiple metrics, each measuring a different failure mode
eval_suite = {
"format_checks": test_json_schema_valid, # Unit test
"faithfulness": ragas_faithfulness_score, # Automated
"task_completion": llm_judge_rubric_score, # Automated
"adversarial": red_team_test_battery, # Automated
"user_satisfaction": human_annotation_sample, # Human (10% sample)
}
# 2. A stable golden dataset that does NOT change between evals
golden_set_path = "eval/golden_v1.json" # Versioned, never edited
# 3. CI gates with minimum thresholds — fail the PR if they drop
ci_gates = {
"format_checks": 1.0, # Zero tolerance for format errors
"faithfulness": 0.85, # Allow 15% headroom for edge cases
"task_completion": 0.80,
}
# 4. Regression tests — catch specific past failures
known_failures = [
"What is 2+2?", # Model once said 5 — fixed, now in suite
"Tell me about GPT-6", # Model once hallucinated specs
]Good evaluation creates a virtuous cycle:
In the next lesson, we survey the benchmark landscape and implement the specific metrics — BLEU, ROUGE, exact match, and task-specific scorers — that belong in your unit eval tier.