Lesson 5: Circuit Breakers, Timeouts & Fallbacks

LLM APIs have outages. Rate limits hit. Timeouts happen. A circuit breaker stops your system from hammering a failing API — and returns a degraded-but-functional response instead of cascading failures that take your entire service down.

The Circuit Breaker State Machine

Closed (Normal)

All requests pass through to the LLM. The breaker tracks failures. If failures exceed threshold within a time window, it opens.

Circuit Breaker Implementation

import time
from enum import Enum
from dataclasses import dataclass, field
import anthropic

client = anthropic.Anthropic()

class BreakerState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

@dataclass
class CircuitBreaker:
    failure_threshold: int = 5        # Open after N failures
    recovery_timeout: float = 60.0    # Seconds before trying again
    half_open_max_calls: int = 1      # Trial calls in half-open state

    _state: BreakerState = field(default=BreakerState.CLOSED, init=False)
    _failure_count: int = field(default=0, init=False)
    _last_failure_time: float = field(default=0.0, init=False)

    def call(self, func, *args, fallback=None, **kwargs):
        """Wrap a function call with circuit breaker protection."""
        if self._state == BreakerState.OPEN:
            if time.monotonic() - self._last_failure_time >= self.recovery_timeout:
                self._state = BreakerState.HALF_OPEN
            else:
                return fallback() if callable(fallback) else fallback

        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            if callable(fallback):
                return fallback()
            raise

    def _on_success(self):
        self._failure_count = 0
        self._state = BreakerState.CLOSED

    def _on_failure(self):
        self._failure_count += 1
        self._last_failure_time = time.monotonic()
        if self._failure_count >= self.failure_threshold:
            self._state = BreakerState.OPEN
            print(f"Circuit OPENED after {self._failure_count} failures")


# Usage
breaker = CircuitBreaker(failure_threshold=3, recovery_timeout=30)

def llm_call(prompt: str) -> str:
    response = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=200,
        messages=[{"role": "user", "content": prompt}],
    )
    return response.content[0].text

def cached_fallback() -> str:
    return "I'm temporarily unavailable. Please try again shortly."

answer = breaker.call(llm_call, "What is Python?", fallback=cached_fallback)

Timeout Configuration

import httpx
import anthropic

# Always set explicit timeouts — no timeout = potentially blocked forever
client = anthropic.Anthropic(
    http_client=httpx.Client(
        timeout=httpx.Timeout(
            connect=5.0,    # Max time to establish connection
            read=30.0,      # Max time between bytes (stream timeout)
            write=5.0,      # Max time to send request
            pool=5.0,       # Max time to get a connection from pool
        )
    )
)

# Async version
async_client = anthropic.AsyncAnthropic(
    http_client=httpx.AsyncClient(
        timeout=httpx.Timeout(connect=5.0, read=30.0, write=5.0, pool=5.0)
    )
)

Fallback Strategies

Cached response

Return the most recent successful response for a similar query. Best for FAQ-style content that rarely changes.

Static fallback

"I'm temporarily unavailable" + retry-after header. Honest and non-confusing for users.

Downgrade model

Try claude-haiku if claude-sonnet fails. Or try a local model (Ollama) if the API is down.

Rule-based response

For narrow tasks (classification, routing), keep a simple if/else logic as the fallback that runs without any LLM.

Test your fallback before you need it.Run a chaos test monthly: inject failures into your LLM client and verify that the fallback activates and users see a graceful response — not a 500.

Worked Example: Tracing the Breaker Through an Outage

Take the breaker above configured with failure_threshold=3 and recovery_timeout=30. Here is what happens as the provider goes down and later recovers (times in seconds):

t=0   call fails            failure_count = 1   state CLOSED
t=2   call fails            failure_count = 2   state CLOSED
t=5   call fails            failure_count = 3   -> OPEN  (last_failure_time = 5)
t=10  call arrives          now - last_failure = 5 < 30  -> fallback instantly, no API call
t=35  call arrives          35 - 5 = 30 >= 30  -> HALF_OPEN, trial call is sent
      trial fails           failure_count = 4 >= 3  -> OPEN again (last_failure_time = 35)
t=50  call arrives          15 < 30  -> fallback instantly
t=65  call arrives          65 - 35 = 30  -> HALF_OPEN, trial call is sent
      trial succeeds        failure_count = 0   -> CLOSED, normal traffic resumes

Why does this matter? Suppose your service handles 10 requests per second and the provider hangs until your 30 s read timeout. By Little's law, the number of requests in flight is arrival rate times time in system: 10 × 30 = 300 stuck connections and workers, all waiting on a dependency that is not going to answer. Your own service then runs out of workers and fails even for endpoints that do not use the LLM. An open breaker turns each of those 30-second waits into an instant fallback, so the failure stays contained.

Retries: Useful but Dangerous

Retries handle brief glitches, but retrying blindly makes outages worse. If 1,000 clients each retry 3 times against a struggling server, the server sees up to 4,000 requests instead of 1,000. Use exponential backoff with random jitter so clients spread out, retry only errors that can succeed on a second try (timeouts, connection errors, rate limits, 5xx), and never retry client errors like a malformed request:

import random
import time

def call_with_retry(func, max_attempts=3, base_delay=1.0, max_delay=8.0):
    for attempt in range(max_attempts):
        try:
            return func()
        except RetryableError:                  # your own class for timeouts, 429, 5xx
            if attempt == max_attempts - 1:
                raise
            delay = min(max_delay, base_delay * 2 ** attempt)   # 1s, 2s, 4s ... capped
            time.sleep(random.uniform(0, delay))                # "full jitter"

Put the retry inside the breaker call (or count only the final failure) so one slow request that retries three times counts as one failure and not three. If the provider returns a Retry-After header on a rate limit, honor it instead of guessing.

Chaining Fallbacks

Fallbacks work best as an ordered chain, from best quality to most basic, with each step guarded separately:

def answer(prompt: str) -> dict:
    steps = [
        ("primary",   lambda: breaker_primary.call(llm_call, prompt, model="primary-model")),
        ("secondary", lambda: breaker_secondary.call(llm_call, prompt, model="smaller-model")),
        ("cache",     lambda: lookup_cached_answer(prompt)),   # returns None on a miss
    ]
    for name, step in steps:
        try:
            result = step()
            if result:
                return {"answer": result, "source": name}
        except Exception:
            continue
    return {"answer": "The assistant is unavailable right now. Please try again in a minute.",
            "source": "static"}

Returning the source lets you monitor how often each step is used and tell users when they received a degraded answer. Use a separate breaker per dependency: one shared breaker would let a failure of the secondary model block the primary.

Limits of the Simple Implementation

Consecutive vs. windowed failures. The class counts failures until a success, not failures in a time window, so five errors spread over a day can open it. Production libraries track a rolling error rate.
Not thread-safe. Counters are updated without a lock. In a threaded or async server, protect state changes or use a library.
Per-process state. Each worker has its own breaker. That is usually acceptable, but a shared store is needed if you want one decision across instances.
Half-open admits many callers. The state check does not limit trial calls to half_open_max_calls; concurrent requests could all become trials.
Which errors count. Count timeouts, connection failures, and 5xx. Do not count 400-level errors caused by your own bad requests, or a code bug will trip the breaker.

Common Pitfalls

Setting a timeout longer than your user will wait. If the page gives up after 10 s, a 30 s read timeout only wastes capacity.
Tripping on rate limits (429). That signals you should slow down, which is different from the provider being down.
A fallback that calls the same failing dependency, or that is itself slow.
No alert when the breaker opens. An open breaker means degraded service and someone should know.
Never exercising the fallback, so it breaks silently. Test it as the callout above says.

Recap

Timeouts bound each call, retries with backoff and jitter absorb brief faults, circuit breakers stop you from hammering a failing provider and exhausting your own workers, and a chain of fallbacks keeps users served in a degraded but honest way. Together they turn a provider outage from your outage into a reduced-quality period.

Lesson 6 moves from resilience to performance: load testing your LLM application to find the breaking points before they hit production.

import time from enum import Enum from dataclasses import dataclass, field import anthropic client = anthropic.Anthropic() class BreakerState(Enum): CLOSED = "closed" OPEN = "open" HALF_OPEN = "half_open" @dataclass class CircuitBreaker: failure_threshold: int = 5 # Open after N failures recovery_timeout: float = 60.0 # Seconds before trying again half_open_max_calls: int = 1 # Trial calls in half-open state _state: BreakerState = field(default=BreakerState.CLOSED, init=False) _failure_count: int = field(default=0, init=False) _last_failure_time: float = field(default=0.0, init=False) def call(self, func, *args, fallback=None, **kwargs): """Wrap a function call with circuit breaker protection.""" if self._state == BreakerState.OPEN: if time.monotonic() - self._last_failure_time >= self.recovery_timeout: self._state = BreakerState.HALF_OPEN else: return fallback() if callable(fallback) else fallback try: result = func(*args, **kwargs) self._on_success() return result except Exception as e: self._on_failure() if callable(fallback): return fallback() raise def _on_success(self): self._failure_count = 0 self._state = BreakerState.CLOSED def _on_failure(self): self._failure_count += 1 self._last_failure_time = time.monotonic() if self._failure_count >= self.failure_threshold: self._state = BreakerState.OPEN print(f"Circuit OPENED after {self._failure_count} failures") # Usage breaker = CircuitBreaker(failure_threshold=3, recovery_timeout=30) def llm_call(prompt: str) -> str: response = client.messages.create( model="claude-haiku-4-5-20251001", max_tokens=200, messages=[{"role": "user", "content": prompt}], ) return response.content[0].text def cached_fallback() -> str: return "I'm temporarily unavailable. Please try again shortly." answer = breaker.call(llm_call, "What is Python?", fallback=cached_fallback)

import httpx import anthropic # Always set explicit timeouts — no timeout = potentially blocked forever client = anthropic.Anthropic( http_client=httpx.Client( timeout=httpx.Timeout( connect=5.0, # Max time to establish connection read=30.0, # Max time between bytes (stream timeout) write=5.0, # Max time to send request pool=5.0, # Max time to get a connection from pool ) ) ) # Async version async_client = anthropic.AsyncAnthropic( http_client=httpx.AsyncClient( timeout=httpx.Timeout(connect=5.0, read=30.0, write=5.0, pool=5.0) ) )

t=0 call fails failure_count = 1 state CLOSED t=2 call fails failure_count = 2 state CLOSED t=5 call fails failure_count = 3 -> OPEN (last_failure_time = 5) t=10 call arrives now - last_failure = 5 < 30 -> fallback instantly, no API call t=35 call arrives 35 - 5 = 30 >= 30 -> HALF_OPEN, trial call is sent trial fails failure_count = 4 >= 3 -> OPEN again (last_failure_time = 35) t=50 call arrives 15 < 30 -> fallback instantly t=65 call arrives 65 - 35 = 30 -> HALF_OPEN, trial call is sent trial succeeds failure_count = 0 -> CLOSED, normal traffic resumes

import random import time def call_with_retry(func, max_attempts=3, base_delay=1.0, max_delay=8.0): for attempt in range(max_attempts): try: return func() except RetryableError: # your own class for timeouts, 429, 5xx if attempt == max_attempts - 1: raise delay = min(max_delay, base_delay * 2 ** attempt) # 1s, 2s, 4s ... capped time.sleep(random.uniform(0, delay)) # "full jitter"

def answer(prompt: str) -> dict: steps = [ ("primary", lambda: breaker_primary.call(llm_call, prompt, model="primary-model")), ("secondary", lambda: breaker_secondary.call(llm_call, prompt, model="smaller-model")), ("cache", lambda: lookup_cached_answer(prompt)), # returns None on a miss ] for name, step in steps: try: result = step() if result: return {"answer": result, "source": name} except Exception: continue return {"answer": "The assistant is unavailable right now. Please try again in a minute.", "source": "static"}

Lesson 5: Circuit Breakers, Timeouts & Fallbacks

The Circuit Breaker State Machine

Circuit Breaker Implementation

Timeout Configuration

Fallback Strategies

Worked Example: Tracing the Breaker Through an Outage

Retries: Useful but Dangerous

Chaining Fallbacks

Limits of the Simple Implementation

Common Pitfalls

Recap

Related articles

The Circuit Breaker State Machine

Circuit Breaker Implementation

Timeout Configuration

Fallback Strategies

Worked Example: Tracing the Breaker Through an Outage

Retries: Useful but Dangerous

Chaining Fallbacks

Limits of the Simple Implementation

Common Pitfalls

Recap