A model cascade routes each request to the cheapest model that can handle it — and automatically escalates to a stronger model when the cheaper one isn't confident enough. A well-tuned cascade cuts API costs by 60–80% with near-zero quality loss.
Short factual queries, classification, extraction, format conversion
# Tier 1 — Fast & cheap (80% of traffic)
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=300,
messages=[{"role": "user", "content": prompt}],
)
# Cost: ~$0.0002/request Latency: ~300msimport anthropic
from pydantic import BaseModel
client = anthropic.Anthropic()
class CascadeResponse(BaseModel):
answer: str
confidence: float # 0.0 - 1.0
escalated: bool
model_used: str
def classify_query_complexity(query: str) -> str:
"""
Fast heuristic routing — before any LLM call.
Avoids even the cheapest model call for obvious cases.
"""
if len(query.split()) < 10 and "?" not in query:
return "simple"
if any(kw in query.lower() for kw in ["analyze", "compare", "explain why", "design"]):
return "complex"
return "medium"
CASCADE = [
("claude-haiku-4-5-20251001", 0.85), # Escalate if confidence < 0.85
("claude-sonnet-4-6", 0.75),
("claude-opus-4-8", 0.0), # Final tier — always trust
]
CONFIDENCE_PROMPT = """Answer the question. Then rate your confidence (0.0-1.0) that your
answer is complete and accurate. Format as JSON: {{"answer": "...", "confidence": 0.XX}}
Question: {question}"""
def cascade_query(question: str) -> CascadeResponse:
initial_tier = classify_query_complexity(question)
start_index = {"simple": 0, "medium": 1, "complex": 2}[initial_tier]
for model, escalation_threshold in CASCADE[start_index:]:
import json
response = client.messages.create(
model=model,
max_tokens=500,
messages=[{"role": "user", "content": CONFIDENCE_PROMPT.format(question=question)}],
)
try:
data = json.loads(response.content[0].text)
confidence = float(data.get("confidence", 0))
if confidence >= escalation_threshold or model == CASCADE[-1][0]:
return CascadeResponse(
answer=data["answer"],
confidence=confidence,
escalated=(model != CASCADE[start_index][0]),
model_used=model,
)
# else: confidence too low — escalate to next tier
except Exception:
# Parse error — escalate
continue
# Should never reach here
raise RuntimeError("Cascade exhausted without response")
result = cascade_query("What is 2 + 2?")
print(f"Model: {result.model_used}, Confidence: {result.confidence:.0%}")
# → Model: claude-haiku-4-5-20251001, Confidence: 99%
result = cascade_query("Analyze the regulatory implications of the EU AI Act for a healthcare AI startup.")
print(f"Model: {result.model_used}, Escalated: {result.escalated}")
# → Model: claude-opus-4-8, Escalated: TrueConfidence-based escalation requires extra LLM calls. A faster approach is task-based routing — classify the query type with a cheap call and route deterministically:
TASK_ROUTER_PROMPT = """Classify this query into one of: simple | medium | complex
Reply with only one word.
simple: factual lookups, format conversion, short extraction
medium: code generation, summarization, RAG Q&A
complex: multi-step analysis, ambiguous judgment, long-form research
Query: {query}"""
def route_by_task(query: str) -> str:
"""One haiku call to route — then use the right tier directly."""
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=10,
messages=[{"role": "user", "content": TASK_ROUTER_PROMPT.format(query=query)}],
)
tier = response.content[0].text.strip().lower()
return {"simple": "claude-haiku-4-5-20251001",
"medium": "claude-sonnet-4-6",
"complex": "claude-opus-4-8"}.get(tier, "claude-sonnet-4-6")A cascade does not make every request cheaper. Requests that escalate pay for the failed cheaper attempt as well as the final one. So the expected cost per request is the sum, over tiers, of the cost of that tier times the probability a request reaches it. Using the illustrative per-request costs from the previous lesson (small $0.000875, mid $0.0105, large $0.0525, which is roughly a 1 : 12 : 60 ratio), suppose measured traffic looks like this: 70% of requests are accepted at the small tier, 20% at the mid tier, and 10% reach the large tier. Every request visits tier 1, 30% visit tier 2, and 10% visit tier 3:
c1, c2, c3 = 0.000875, 0.0105, 0.0525
reach2, reach3 = 0.30, 0.10 # share of traffic that reaches tiers 2 and 3
expected = c1 + reach2 * c2 + reach3 * c3
# 0.000875 + 0.30 * 0.0105 + 0.10 * 0.0525
# = 0.000875 + 0.00315 + 0.00525 = 0.009275
print(expected / c3) # 0.177 -> 82% cheaper than sending everything to the large model
print(expected / c2) # 0.883 -> only 12% cheaper than sending everything to the mid modelThe second line is the important one. Against an all-large baseline the cascade looks excellent, but against a well-chosen single mid-tier model it saves little, and the cascade adds extra latency for every escalated request. The headline savings in cascade write-ups depend heavily on the baseline they compare with.
For a two-tier cascade there is a clean break-even rule. If the cheap model is accepted with probability a, cost is c₁ + (1 − a)·c₂, compared with c₂ for always using the stronger model. The cascade wins when c₁ < a·c₂, that is, when a > c₁/c₂. With c₁/c₂ = 0.000875 / 0.0105 ≈ 0.083, the cheap tier only needs to be accepted on about 8% of traffic to pay for itself in dollars. In practice, the tighter constraint is quality: the accepted answers must really be as good as the stronger model's.
A threshold such as 0.85 is only meaningful if the confidence number is. A model's self-reported confidence is often poorly calibrated, typically too high, so do not trust it until you have checked it. Collect a labeled sample, record the confidence for each answer, and measure accuracy among answers accepted at each threshold:
# rows: (confidence, was_correct) from a labeled evaluation set
def threshold_report(rows, thresholds=(0.6, 0.7, 0.8, 0.9)):
for t in thresholds:
accepted = [ok for conf, ok in rows if conf >= t]
if not accepted:
continue
coverage = len(accepted) / len(rows) # share answered by the cheap tier
accuracy = sum(accepted) / len(accepted) # quality of what it answered
print(f"threshold {t:.1f}: coverage {coverage:.0%}, accuracy {accuracy:.1%}")Raising the threshold increases accuracy of accepted answers but lowers coverage, so more traffic escalates. Pick the lowest threshold whose accuracy meets your quality bar. Stronger signals than a self-rating include log probabilities where an API provides them, agreement between several samples, a schema validator, or a separate verifier model. Cheap programmatic checks (valid JSON, required fields present, answer found in the retrieved context) are often the most reliable.
str.format on a prompt that contains literal JSON raises a KeyError. Double the braces ({{ and }}) or use a different templating method.A cascade trades extra calls on hard requests for savings on easy ones. Compute expected cost from measured tier acceptance rates, compare against the best single model rather than the most expensive one, calibrate your thresholds on labeled data, and monitor escalation rate and quality in production.
In the next lesson, we add semantic caching on top of the cascade — so repeated similar queries never reach the LLM at all.