Offline eval suites test snapshots. Production evals test the real thing — with real users, real edge cases, and real model drift. Running evals continuously on live traffic is the highest-signal quality signal you have.
Offline evals run on a curated golden set. Online evals run on real traffic. They measure different things:
| Dimension | Offline (Golden Set) | Online (Production) |
|---|---|---|
| Coverage | Known failure modes | Unknown distribution — the long tail |
| Cost | Fixed budget per run | Ongoing — sample to control cost |
| Latency | Acceptable to be slow | Must not affect p99 of user requests |
| Cadence | Per commit / per PR | Continuous — 24/7 |
| Catch | Regressions from code changes | Drift from model updates, data distribution shift |
The sample rate controls your cost vs. detection speed trade-off:
Balanced default. Catches drift within hours. Affordable for most production systems.
SAMPLE_RATE = 0.10 # 10% of requests — recommended default
async def shadow_eval(request_id, user_question, response, latency_ms):
if random.random() > SAMPLE_RATE:
return None
# At 10,000 req/day → 1,000 evals/day
# At $0.0025 per eval (haiku): $2.50/day eval cost
# Coverage: catches model drift within 2-4 hours
# Sweet spot for most production AI applications
score = await judge_response(user_question, response)
await write_metric(request_id, score, latency_ms)# eval_pipeline.py — shadow eval on 10% of production traffic
import anthropic
import random
import asyncio
from datetime import datetime
from dataclasses import dataclass, asdict
import json
client = anthropic.Anthropic()
@dataclass
class EvalRecord:
request_id: str
user_question: str
model_response: str
quality_score: float
timestamp: str
latency_ms: int
SAMPLE_RATE = 0.10 # Eval 10% of requests
async def shadow_eval(
request_id: str,
user_question: str,
model_response: str,
latency_ms: int,
) -> EvalRecord | None:
"""Run async in the background — do NOT block the user response."""
if random.random() > SAMPLE_RATE:
return None # Not selected for eval this time
try:
judge = client.messages.create(
model="claude-haiku-4-5-20251001", # Cheap fast judge
max_tokens=50,
messages=[{
"role": "user",
"content": f"""Score this AI response 1-10.
Question: {user_question}
Response: {model_response}
Output only the number.""",
}],
)
score = float(judge.content[0].text.strip()) / 10
record = EvalRecord(
request_id=request_id,
user_question=user_question,
model_response=model_response,
quality_score=score,
timestamp=datetime.utcnow().isoformat(),
latency_ms=latency_ms,
)
# Write to your metrics store (BigQuery, ClickHouse, Postgres, etc.)
write_to_metrics(asdict(record))
return record
except Exception as e:
# NEVER let eval errors propagate to users
print(f"[eval] Error: {e}")
return Noneimport time
import asyncio
import uuid
from fastapi import FastAPI, BackgroundTasks
app = FastAPI()
@app.post("/chat")
async def chat(request: ChatRequest, background_tasks: BackgroundTasks):
request_id = str(uuid.uuid4())
t0 = time.monotonic()
# ─── Serve the user ────────────────────────────────────────
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
messages=[{"role": "user", "content": request.message}],
)
answer = response.content[0].text
latency_ms = int((time.monotonic() - t0) * 1000)
# ─── Shadow eval in background — user does NOT wait ────────
background_tasks.add_task(
shadow_eval,
request_id=request_id,
user_question=request.message,
model_response=answer,
latency_ms=latency_ms,
)
return {"answer": answer, "request_id": request_id}# monitor.py — runs every 30 minutes via cron
def check_quality_trend(window_hours: int = 4, alert_threshold: float = 0.1) -> None:
"""Alert if rolling average quality drops by 10% vs previous window."""
current_avg = get_avg_quality(hours=window_hours)
previous_avg = get_avg_quality(hours=window_hours * 2, offset=window_hours)
delta = current_avg - previous_avg
if delta < -alert_threshold:
send_alert(
title="AI Quality Degradation Alert",
message=f"Quality dropped from {previous_avg:.2f} to {current_avg:.2f} ({delta:+.2f}) in the last {window_hours}h",
severity="high",
)
# Grafana dashboard query (SQL)
SELECT
DATE_TRUNC('hour', timestamp) AS hour,
AVG(quality_score) AS avg_quality,
COUNT(*) AS eval_count,
PERCENTILE_CONT(0.05) WITHIN GROUP (ORDER BY quality_score) AS p5_quality
FROM eval_records
WHERE timestamp > NOW() - INTERVAL '7 days'
GROUP BY 1
ORDER BY 1Model drift is real.Cloud LLM providers update their models over time. Claude Haiku and Sonnet both received capability updates in 2025 that changed behavior on some tasks. Your production eval dashboard will catch these — your offline suite won't.
You have now covered the complete evaluation lifecycle: from unit tests to LLM judges to production monitoring. The capstone pulls everything together into a deployable eval suite.