LLM quality can degrade silently: a model update, a changed prompt, a shift in user query distribution — none of these raise an error. Quality drift detection catches the degradation before users do.
Best for: Capture a random sample of live requests and score them with an LLM judge. Continuous monitoring at low cost.
import anthropic
import random
import asyncio
client = anthropic.Anthropic()
SAMPLE_RATE = 0.05 # Score 5% of production traffic
async def maybe_score_response(
user_query: str,
llm_response: str,
metadata: dict,
) -> None:
"""Non-blocking: doesn't add latency to user-facing requests."""
if random.random() > SAMPLE_RATE:
return # Skip — not sampled this time
# Run asynchronously so the user response has already been sent
score = await score_response_quality(user_query, llm_response)
write_quality_record({
"query_hash": hash(user_query),
"score": score,
"model": metadata.get("model"),
"operation": metadata.get("operation"),
"timestamp": "ISO8601",
})
async def score_response_quality(query: str, response: str) -> float:
judge = await client.messages.acreate(
model="claude-haiku-4-5-20251001",
max_tokens=10,
messages=[{
"role": "user",
"content": f"""Score this AI response 1-10 (helpfulness, accuracy, completeness).
Reply with just the number.
Question: {query[:300]}
Response: {response[:500]}""",
}],
)
try:
return float(judge.content[0].text.strip()) / 10
except ValueError:
return 0.5 # Default if judge gives unexpected outputNightly regression test against golden dataset. Most providers release updated models on a schedule.
Track the embedding centroid of incoming queries. Large drift = users asking different things than your system was tuned for.
Monitor % of queries with retrieved context older than X days. Stale context → stale answers.
Version control prompts. Run A/B test before and after any prompt change. Enforce regression gate in CI.
Don't use a single quality score. A mean score of 0.85 can hide a bimodal distribution where 70% of queries score 0.95 and 30% score 0.65. Always look at the full score distribution — histogram and p10/p25/p75/p90 percentiles — not just the mean.
Lesson 5 turns your quality monitoring into formal SLOs — Service Level Objectives — giving you a contractual quality target with an automated alert when you breach it.