Different tasks call for different metrics. Using BLEU to evaluate a customer support bot is like measuring a chef's skill by how many ingredients they used. This lesson maps each metric to the tasks it actually measures well.
# pip install rouge-score bert-score sacrebleu
import re
from rouge_score import rouge_scorer
from bert_score import score as bert_score
import sacrebleu
def normalize(text: str) -> str:
"""Lower, strip punctuation, collapse whitespace."""
text = text.lower().strip()
text = re.sub(r"[^ws]", "", text)
text = re.sub(r"s+", " ", text)
return text
def exact_match(prediction: str, reference: str) -> float:
return float(normalize(prediction) == normalize(reference))
def bleu(prediction: str, reference: str) -> float:
result = sacrebleu.sentence_bleu(prediction, [reference])
return result.score / 100 # sacrebleu returns 0–100
def rouge_l(prediction: str, reference: str) -> float:
scorer = rouge_scorer.RougeScorer(["rougeL"], use_stemmer=True)
return scorer.score(reference, prediction)["rougeL"].fmeasure
def bertscore_f1(predictions: list[str], references: list[str]) -> list[float]:
"""BERTScore requires batching — more efficient on lists."""
P, R, F1 = bert_score(predictions, references, lang="en", verbose=False)
return F1.tolist()
# Run on a test set
test_pairs = [
("The capital of France is Paris.", "Paris is the capital of France."),
("The model uses backpropagation.", "Training uses backpropagation."),
]
for pred, ref in test_pairs:
print(f"EM: {exact_match(pred, ref):.2f}")
print(f"BLEU: {bleu(pred, ref):.2f}")
print(f"ROUGE-L:{rouge_l(pred, ref):.2f}")| Task | Recommended Metrics |
|---|---|
| Q&A (factoid) | Exact Match + BERTScore (one right answer) |
| Summarization | ROUGE-L + BERTScore + human eval sample |
| RAG systems | RAGAS (faithfulness, relevancy, precision, recall) |
| Code generation | pass@k (functional correctness via unit tests) |
| Classification | Accuracy, F1, confusion matrix |
| Open-ended chat | LLM-as-judge (no reference answer exists) |
| Translation | BLEU + ChrF (handles multilingual n-grams better) |
def pass_at_k(num_correct: int, num_samples: int, k: int) -> float:
"""
Probability that at least one of k samples passes all unit tests.
Used in HumanEval, MBPP, and other code generation benchmarks.
num_correct: number of samples that passed all tests
num_samples: total samples generated (n)
k: how many samples the model is allowed to use
"""
if num_correct == 0:
return 0.0
if num_samples - num_correct < k:
return 1.0 # fewer than k failures exist, so any k samples include a pass
# Combinatorial formula from Chen et al. (2021)
from math import comb
return 1.0 - comb(num_samples - num_correct, k) / comb(num_samples, k)
# Example: generate 10 solutions, 3 pass tests
# How likely is the model to have at least 1 passing solution if we pick k=3?
print(pass_at_k(num_correct=3, num_samples=10, k=3)) # → 0.7083 (= 1 - C(7,3)/C(10,3) = 1 - 35/120)Exact match is all-or-nothing. A common middle ground for question answering is token-level F1, which gives partial credit for overlapping words. Precision is the share of predicted tokens that appear in the reference; recall is the share of reference tokens that appear in the prediction; F1 is their harmonic mean. Counting duplicates matters, so use a multiset intersection:
from collections import Counter
def token_f1(prediction: str, reference: str) -> float:
pred = normalize(prediction).split()
ref = normalize(reference).split()
overlap = sum((Counter(pred) & Counter(ref)).values())
if overlap == 0:
return 0.0
precision = overlap / len(pred)
recall = overlap / len(ref)
return 2 * precision * recall / (precision + recall)
# pred tokens: the cat sat on the mat (6)
# ref tokens: a cat sat on a mat (6)
# shared: cat, sat, on, mat = 4 -> P = 4/6, R = 4/6, F1 = 0.667
print(round(token_f1("The cat sat on the mat", "A cat sat on a mat"), 3)) # 0.667Many QA benchmarks also strip articles (a, an, the) during normalization. Whatever rule you choose, apply the same one to predictions and references, and write it down, because the normalization alone can move scores noticeably.
ROUGE-L finds the longest common subsequence (LCS) of words, which keeps order but allows gaps. For the prediction "the cat sat on the mat" and the reference "the cat lay on the mat", the LCS is "the cat on the mat", length 5. Precision is 5/6, recall is 5/6, so the F-measure is about 0.83. Replace the reference with a paraphrase that shares few words and the score collapses even if the meaning is identical, which is the weakness the interactive panel above demonstrates.
Choose the metric from the task: exact match or token F1 for short factual answers, ROUGE for summaries, BLEU and chrF for translation, unit tests (pass@k) for code, and embeddings or judges for meaning. No single number is enough, so combine automatic metrics with a human-reviewed sample, report uncertainty, and keep a private test set.
In the next lesson, we build the most powerful tool in the eval arsenal: LLM-as-judge — using a strong model to score another model's outputs against a rubric you define.