AI System Design for Engineers • Module A: Architecture PatternsLesson 1: Model Selection — Capability, Cost & Latency Trade-offs
Picking a model is not a one-time decision — it is an architecture decision that affects latency, cost, quality, and maintainability for the lifetime of your system. This lesson gives you a repeatable framework instead of guesswork.
Every model selection involves trading off three axes. You can optimize for two, but rarely all three:
Capability (quality)
▲
│
claude-sonnet ──┤── claude-opus
│
claude-haiku ──────┤
│
gpt-4o-mini ────┘
◄────────────────►
Cost & Latency (lower = better)
Rule: Pick the cheapest/fastest model that meets your quality bar.
Test if the cheaper model meets quality — don't assume the expensive one is necessary.Select your task type to see the recommended model and a code snippet:
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=50,
messages=[{
"role": "user",
"content": f"Classify this query as: billing, technical, general\n\n{query}"
}],
)
category = response.content[0].text.strip()import anthropic
client = anthropic.Anthropic()
MODELS_TO_TEST = [
"claude-haiku-4-5-20251001", # Test cheapest first
"claude-sonnet-4-6",
# "claude-opus-4-8", # Only if sonnet fails
]
def evaluate_model_for_task(model: str, test_cases: list[dict]) -> float:
"""Score a model on your specific task. Pick the cheapest that passes."""
scores = []
for case in test_cases:
response = client.messages.create(
model=model,
max_tokens=500,
messages=[{"role": "user", "content": case["prompt"]}],
)
score = judge_output(response.content[0].text, case["expected"])
scores.append(score)
return sum(scores) / len(scores)
QUALITY_THRESHOLD = 0.90 # 90% of test cases must pass
for model in MODELS_TO_TEST:
score = evaluate_model_for_task(model, test_cases)
print(f"{model}: {score:.1%}")
if score >= QUALITY_THRESHOLD:
print(f"→ USE THIS MODEL: {model}")
break # Stop — cheapest model that meets quality bar
# → claude-haiku-4-5-20251001: 93.2%
# → USE THIS MODEL: claude-haiku-4-5-20251001# Approximate pricing (verify at anthropic.com/pricing — changes frequently)
PRICING = {
"claude-haiku-4-5-20251001": {"input": 0.00025, "output": 0.00125}, # per 1K tokens
"claude-sonnet-4-6": {"input": 0.003, "output": 0.015},
"claude-opus-4-8": {"input": 0.015, "output": 0.075},
}
def estimate_monthly_cost(
model: str,
avg_input_tokens: int,
avg_output_tokens: int,
requests_per_day: int,
) -> float:
p = PRICING[model]
cost_per_request = (
avg_input_tokens / 1000 * p["input"]
+ avg_output_tokens / 1000 * p["output"]
)
return cost_per_request * requests_per_day * 30
# Example: RAG app, 10K requests/day
for model in PRICING:
monthly = estimate_monthly_cost(model, 2000, 300, 10_000)
print(f"{model}: ${monthly:,.0f}/month")
# → claude-haiku-4-5-20251001: $262/month
# → claude-sonnet-4-6: $3,150/month
# → claude-opus-4-8: $15,750/monthAlmost every AI system has tasks of different complexity. Classify them: use haiku for routing and extraction, sonnet for generation, and reserve opus (or a domain fine-tune) for the small fraction of genuinely hard cases. The cost difference between tiers is 10–60× — the quality difference is usually much smaller.
Take the RAG app from the cost script above: 10,000 requests a day, 2,000 input tokens and 300 output tokens per request. Using the illustrative prices in that table, the per-request cost on the small model is 2 × $0.00025 + 0.3 × $0.00125 = $0.0005 + $0.000375 = $0.000875. On the mid-tier model it is 2 × $0.003 + 0.3 × $0.015 = $0.006 + $0.0045 = $0.0105. Over 300,000 requests a month that is about $262 versus $3,150.
Now suppose your evaluation shows that 80% of requests (simple lookups) pass your quality bar on the small model, and only 20% need the mid-tier one. The blended cost is 0.8 × $262.50 + 0.2 × $3,150 = $210 + $630 = $840 a month, a 73% reduction against sending everything to the mid-tier model, with the quality bar still met. The 80/20 split is a made-up number for the arithmetic; you must measure yours on your own traffic. Prices change, so always recompute with the current price list.
The quality protocol above says "93.2%, use this model". With how many test cases? A score is an estimate with uncertainty. For a pass rate p measured on n cases, the standard error is √(p(1−p)/n). With p = 0.932 and n = 100, that is √(0.932 × 0.068 / 100) = √0.000634 = 0.025, so a 95% interval is about ±1.96 × 2.5 = ±4.9 points, or roughly 88% to 98%. That interval includes your 90% threshold, so 100 cases cannot tell you the cheap model really passes. With n = 500 the standard error is 0.0113 and the interval narrows to about ±2.2 points (91% to 95%).
import math
def pass_rate_interval(passes: int, n: int, z: float = 1.96):
p = passes / n
se = math.sqrt(p * (1 - p) / n)
return p - z * se, p + z * se
print(pass_rate_interval(93, 100)) # about (0.880, 0.980)
print(pass_rate_interval(466, 500)) # about (0.910, 0.954)Build the test set from real production inputs, include the hard and unusual cases on purpose, and compare models on the same cases so differences reflect the model and not the sample.
Response time is roughly the time to first token plus the output length divided by generation speed. With hypothetical figures of 0.4 s to first token and 100 tokens per second, a 300-token answer takes about 0.4 + 300 / 100 = 3.4 s, while a 20-token classification takes 0.4 + 0.2 = 0.6 s. Output length often dominates, so asking for a short structured answer, setting a sensible max_tokens, and streaming the reply to the user can help more than switching models. Measure p95 latency under realistic load, not a single timed call.
Start from your quality bar and latency budget, test the cheapest candidate first on a large enough set of real cases, estimate cost with your actual token counts, and tier tasks so expensive models only handle what needs them. Treat the choice as reversible and keep the evaluation around so you can revisit it.
In the next lesson, we implement this tiering mechanically as a model cascade — a system that routes each request to the cheapest model likely to handle it well, and automatically escalates when confidence is low.