Building Reliable LLM Applications: Rate Limiting, Retries, and Fallbacks

When building applications that rely on external Large Language Model (LLM) APIs, developers must design for instability. Unlike traditional SaaS APIs, LLM endpoints are prone to high latency, transient timeouts, sudden rate limits (429 Too Many Requests), and occasional service outages.

If your application calls these APIs synchronously without a resilience layer, users will experience frequent errors. To build production-grade, reliable AI systems, you must implement three resilience patterns: Exponential Backoff with Jitter, Model Fallbacks, and Token-Aware Rate Limiting.

---

1. Retries with Exponential Backoff and Jitter

When an API call fails due to a rate limit or a transient server error (5\text{xx}), retrying immediately can overload the destination server and exacerbate the issue. Instead, implement Exponential Backoff.

Exponential backoff increases the delay between retries exponentially:

t_{\text{delay}} = t_{\text{base}} \times 2^{\text{attempt}}

However, if multiple client requests fail at the same time, they will retry at the exact same intervals, causing a synchronized spike in traffic known as the thundering herd problem. To prevent this, we introduce Jitter—a randomized offset added to the backoff calculation:

t_{\text{delay\_with\_jitter}} = \text{random}(0, t_{\text{delay}})

  Attempt 1: [Error 429] ---> Wait 1s + Jitter (0.4s)
  Attempt 2: [Error 429] ---> Wait 2s + Jitter (1.2s)
  Attempt 3: [Error 429] ---> Wait 4s + Jitter (3.1s)
  Attempt 4: [Success]

---

2. Model Fallbacks (Failover Routing)

No single AI provider guarantees 100% uptime. To protect your application from provider outages, configure Model Fallbacks. If your primary model (e.g., gpt-4o) fails repeatedly or times out, the system should catch the exception and route the request to a secondary provider (e.g., Anthropic's claude-3-5-sonnet or a local model hosted on an inference provider).

Building Reliable LLM Applications: Reliability framework diagram showing API gateway retry with backoff, rate limiting, and fallback routing

---

3. Client-Side Rate Limiting

Providers enforce two types of rate limits: Requests Per Minute (RPM) and Tokens Per Minute (TPM).

Instead of waiting for a 429 error to occur, you can track token consumption client-side using a Token Bucket algorithm. Before sending a prompt, check if the bucket contains enough capacity for both the input tokens and the estimated output tokens. If the capacity is exceeded, queue the request until the bucket refills.

---

Python Implementation: Resilient API Client

Below is a complete Python script demonstrating how to construct a resilient LLM wrapper utilizing the tenacity retry library to handle exponential backoff with jitter, paired with a custom failover system to catch API failures and fall back to alternative models.

import os
import random
import time
from openai import OpenAI, APIConnectionError, RateLimitError, APIStatusError
from tenacity import retry, wait_random_exponential, stop_after_attempt, retry_if_exception_type

# Initialize multiple clients (API keys must be present in environment variables)
openai_client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "mock-key"))
# We simulate a fallback system below; normally this might be Anthropic or Azure OpenAI

PRIMARY_MODEL = "gpt-4o"
FALLBACK_MODEL = "gpt-4o-mini"

# 1. Define Retry Logic for Transient Failures
# This decorator will retry the function if it encounters rate limits or connection errors.
# It uses exponential backoff with random jitter up to 10 seconds, stopping after 4 attempts.
@retry(
    wait=wait_random_exponential(min=1, max=10),
    stop=stop_after_attempt(4),
    retry=retry_if_exception_type((RateLimitError, APIConnectionError)),
    reraise=True  # Reraise exception if all attempts fail so we can trigger the fallback
)
def call_primary_llm(prompt: str) -> str:
    print(f"[{time.strftime('%H:%M:%S')}] Attempting primary model: {PRIMARY_MODEL}...")
    
    # Simulate API errors for demonstration purposes if keys are mocked
    if os.environ.get("OPENAI_API_KEY") is None:
        if random.random() < 0.7:  # 70% chance of failure to trigger retries
            print("  (Simulating transient RateLimitError)")
            raise RateLimitError(
                message="Rate limit exceeded", 
                response=None, 
                body=None
            )
            
    response = openai_client.chat.completions.create(
        model=PRIMARY_MODEL,
        messages=[{"role": "user", "content": prompt}],
        timeout=15.0  # Set strict timeouts to avoid hanging requests
    )
    return response.choices[0].message.content

# 2. Define the Orchestration Function with Failover
def generate_text_with_resilience(prompt: str) -> str:
    try:
        # Try calling the primary model (handled by tenacity retry logic)
        return call_primary_llm(prompt)
    except Exception as e:
        print(f"\n[WARNING] Primary model failed after maximum retries. Error: {e}")
        print(f"Switching to fallback model: {FALLBACK_MODEL}...")
        
        try:
            # Call the fallback model (with direct execution or a simpler retry config)
            response = openai_client.chat.completions.create(
                model=FALLBACK_MODEL,
                messages=[{"role": "user", "content": prompt}],
                timeout=10.0
            )
            print("[SUCCESS] Fallback request resolved successfully.")
            return response.choices[0].message.content
        except Exception as fallback_err:
            # Both primary and fallback failed; raise a clean error to the application
            print(f"[FATAL] Fallback model also failed: {fallback_err}")
            raise RuntimeError("All LLM providers are currently unavailable. Please try again later.")

# --- Demo Run ---
if __name__ == "__main__":
    prompt_query = "Summarize the architectural principles of microservices."
    try:
        result = generate_text_with_resilience(prompt_query)
        print(f"\nFinal Result Output:\n{result[:150]}...")
    except Exception as fatal_error:
        print(f"\nExecution Failed: {fatal_error}")

---

Best Practices Checklist

To build reliable systems, ensure your pipeline respects the following rules:

• Always Set Timeouts: Default client connections can hang for minutes. Always pass a strict timeout parameter (e.g., 10 to 15 seconds) to prevent cascading delays in your backend threads.
• Monitor Headers: Parse response headers like x-ratelimit-remaining-tokens and x-ratelimit-reset-requests to dynamically adjust your client's request rate.
• Alert on Failover: Set up monitoring alerts when fallbacks are triggered. If fallback models are active, it indicates that either your main API quota has expired, or the primary provider is experiencing an outage.
• Graceful Degradation: If an LLM call fails, return cached answers, simplified defaults, or pre-written mock templates instead of showing an empty screen.