When no reference answer exists, use a stronger LLM to judge a weaker one. LLM-as-judge scales human-level evaluation to thousands of outputs per hour — at a fraction of annotation cost.
Strong frontier models (claude-sonnet-4-6, GPT-4o) show 80–90% agreement with human expert annotators on structured rubrics — comparable to inter-human agreement. This makes them practical judges for open-ended generation tasks where exact-match metrics fail.
A rubric converts a fuzzy quality dimension into a 1–5 scale the model can apply consistently. Click a dimension to see how a rubric is structured:
“Python was created by Guido van Rossum and released in 1991. It uses indentation to define code blocks and is widely used in data science.”
All three stated facts are correct: creator (Guido van Rossum), release year (1991), and the indentation-based syntax.
import anthropic
import json
from pydantic import BaseModel
client = anthropic.Anthropic()
class JudgeOutput(BaseModel):
score: int # 1–5
reasoning: str # Explanation of the score
improvement: str # Suggested improvement
JUDGE_PROMPT = """You are an expert evaluator assessing the quality of AI-generated responses.
## Evaluation Rubric: {dimension}
{rubric}
## Question Asked
{question}
## Response to Evaluate
{response}
## Instructions
1. Apply the rubric carefully
2. Provide your score (1–5), reasoning (2–3 sentences), and one specific improvement suggestion
3. Output ONLY valid JSON matching this schema: {schema}"""
RUBRICS = {
"accuracy": """
5 — All claims are correct and verifiable
4 — Minor inaccuracies that do not affect core meaning
3 — Some factual errors but overall direction is correct
2 — Major factual errors present
1 — Largely incorrect or hallucinates""",
"completeness": """
5 — Comprehensively addresses the question with no gaps
4 — Addresses main points, minor gaps
3 — Covers some aspects, misses important points
2 — Superficial coverage, major gaps
1 — Barely addresses the question""",
"clarity": """
5 — Clear, well-organized, easy to follow
4 — Mostly clear with minor structure issues
3 — Understandable but disorganized or verbose
2 — Difficult to follow
1 — Incomprehensible""",
}
SCHEMA = '{"score": int, "reasoning": str, "improvement": str}'
def llm_judge(question: str, response: str, dimension: str = "accuracy") -> JudgeOutput:
prompt = JUDGE_PROMPT.format(
dimension=dimension,
rubric=RUBRICS[dimension],
question=question,
response=response,
schema=SCHEMA,
)
result = client.messages.create(
model="claude-sonnet-4-6", # Use a strong model as judge
max_tokens=512,
messages=[{"role": "user", "content": prompt}],
)
data = json.loads(result.content[0].text)
return JudgeOutput(**data)
# Multi-dimensional eval
def eval_response(question: str, response: str) -> dict[str, JudgeOutput]:
return {dim: llm_judge(question, response, dim) for dim in RUBRICS}
result = eval_response(
question="What is gradient descent?",
response="Gradient descent is a first-order iterative optimization algorithm for finding local minima.",
)
for dim, judge in result.items():
print(f"{dim}: {judge.score}/5 — {judge.reasoning[:60]}...")When comparing two responses A vs B, also run B vs A and average — judges prefer the first option at ~60% rate
Longer answers score higher even when less accurate. Add "length should not affect your score" to the prompt
A model judging its own outputs inflates scores ~15%. Use a different model as judge than the one being evaluated
Use temperature=0 for the judge to reduce score variance across identical inputs
In the next lesson, we move from evaluating outputs to testing the full application: unit tests that assert LLM behavior deterministically, even for non-deterministic functions.