Rate limits are a shared resource problem: your application competes for API capacity with itself (multiple threads, users, microservices) and the provider's other customers. Client-side rate limiting prevents you from hitting provider limits and gives you per-tenant cost control.
import time, threading
from dataclasses import dataclass, field
@dataclass
class TokenBucket:
capacity: int # Max burst size (e.g., 10 requests)
rate: float # Refill rate (e.g., 50 per 60s = 0.833/s)
_tokens: float = field(init=False)
_last_refill: float = field(default_factory=time.monotonic, init=False)
_lock: threading.Lock = field(default_factory=threading.Lock, init=False)
def __post_init__(self):
self._tokens = float(self.capacity)
def acquire(self, timeout: float = 30.0) -> bool:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
with self._lock:
elapsed = time.monotonic() - self._last_refill
self._tokens = min(self.capacity, self._tokens + elapsed * self.rate)
self._last_refill = time.monotonic()
if self._tokens >= 1:
self._tokens -= 1
return True
time.sleep(1.0 / self.rate)
return False
# 50 RPM, allow burst of 10
rpm_limiter = TokenBucket(capacity=10, rate=50/60)
def call_llm(prompt: str) -> str:
if not rpm_limiter.acquire(timeout=60):
raise TimeoutError("Could not acquire RPM slot within 60s")
return client.messages.create(...).content[0].textRequests vary wildly in cost. A 1-token request and a 10K-token request both count as 1 RPM but have very different costs. Track input+output tokens for meaningful quota enforcement.
Alert tenants at 80% of quota usage. Hard-cut at 100%. A surprised user who was never warned is an angry user — give them the chance to upgrade.
Return X-Quota-Used: 45000, X-Quota-Limit: 50000 in every response. Clients can implement their own throttling before hitting the hard limit.
Pre-flight token estimation prevents quota overshoot.Estimate input tokens before the call using len(prompt) / 4(rough approximation). If the estimate would exceed quota, reject early rather than discovering overshoot after the API call — you've already paid for those tokens.
The RPM limiter above has capacity 10 and a refill rate of 50/60 = 0.833 tokens per second. Suppose 25 requests arrive at the same instant with a full bucket. The first 10 consume the bucket and go out immediately. The remaining 15 must wait for refills, and since one token arrives every 1 / 0.833 = 1.2 seconds, they finish going out over 15 × 1.2 = 18 seconds. Over any long period the average is capped at 50 per minute, but a short burst of up to 10 is allowed. Capacity therefore controls how bursty you can be, and the refill rate controls the sustained throughput. If the provider enforces its limit in strict one-minute windows, keep the capacity small so that bursts do not collide with the window edge.
Several limits apply at once and the lowest effective one wins. With the illustrative limits of 50 RPM and 50,000 TPM, a workload averaging 2,500 tokens per request (input plus output) is capped by tokens, not requests:
rpm_limit, tpm_limit = 50, 50_000
avg_tokens = 2_500
by_requests = rpm_limit # 50 requests/min
by_tokens = tpm_limit // avg_tokens # 50,000 / 2,500 = 20 requests/min
print(min(by_requests, by_tokens)) # 20 -> TPM is the real ceiling
# If you shrink the average request to 800 tokens:
print(min(rpm_limit, tpm_limit // 800)) # min(50, 62) = 50 -> now RPM bindsSo trimming prompts or lowering max_tokens can raise throughput more than requesting a higher request limit. Check your own limits in the provider console, since they depend on your account tier and change over time.
A semaphore caps how many requests are in flight, but throughput depends on latency. By Little's law, concurrency = throughput × average latency. To sustain 50 requests per minute (0.833 per second) when the average call takes 4 seconds, you need about 0.833 × 4 = 3.3 concurrent requests, so a limit of 5 is enough to reach the rate limit. Conversely, with a semaphore of 5 and 4-second calls, the maximum throughput is 5 / 4 = 1.25 requests per second, which is 75 per minute. The semaphore alone would not keep you under 50 RPM, so combine it with the token bucket.
A call's token use is the input plus the output, and you do not know the output length in advance. A safe pre-flight estimate reserves the worst case, then corrects the account with the real usage afterward:
def call_with_reservation(prompt: str, max_tokens: int = 500):
reserved = len(prompt) // 4 + max_tokens # rough input estimate + worst-case output
if not tpm_limiter.acquire(reserved):
raise RuntimeError("Rate limited locally; retry later")
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=max_tokens,
messages=[{"role": "user", "content": prompt}],
)
actual = response.usage.input_tokens + response.usage.output_tokens
tpm_limiter.record_adjustment(actual - reserved) # you implement: refund or charge the difference
return response.content[0].textThe character-count rule of thumb of about four characters per token is only an approximation for English text; code and other languages tokenize differently. For accurate counts use the provider's token-counting facility and the usage field returned with each response.
Client-side limiting reduces 429 errors but cannot remove them, because other processes or the provider's own load can still push you over. When one arrives, read the Retry-After header if present and wait that long, add random jitter so many workers do not wake simultaneously, and cap the number of retries. Treat repeated 429s as a signal to slow the whole system down, for example by lowering the bucket rate, instead of letting each request retry independently.
If one customer can consume the whole shared limit, all others suffer. Give each tenant its own bucket and daily counter, sized as a share of the global capacity. For example, with a global 50 RPM and 5 equal tenants, each gets 10 RPM. Since tenants rarely peak together, you can allow borrowing of unused capacity up to a cap, while keeping a floor reserved so that every tenant can always make progress.
INCRBY, as in the daily quota example, or a Lua script.Understand which limit binds for your workload, shape bursts with a token bucket, bound in-flight work with a semaphore, reserve tokens before each call and reconcile afterward, back off with jitter on 429, and give each tenant its own budget so one heavy user cannot starve the rest.