An SLO (Service Level Objective) is a formal quality target with a measurement method and an alert when you breach it. For AI systems, SLOs extend beyond uptime and latency to cover quality metrics that traditional infrastructure doesn't have.
from collections import deque
import time
class AvailabilityTracker:
def __init__(self, window_seconds: int = 300, target: float = 0.995):
self.window = window_seconds
self.target = target
self._events: deque[tuple[float, bool]] = deque()
def record(self, success: bool):
now = time.monotonic()
self._events.append((now, success))
cutoff = now - self.window
while self._events and self._events[0][0] < cutoff:
self._events.popleft()
def current_rate(self) -> float:
if not self._events:
return 1.0
return sum(1 for _, ok in self._events if ok) / len(self._events)
def is_breached(self) -> bool:
return self.current_rate() < self.target
tracker = AvailabilityTracker(window_seconds=300, target=0.995)
try:
response = llm_call(prompt)
tracker.record(success=True)
except Exception:
tracker.record(success=False)
if tracker.is_breached():
send_alert(f"Availability SLO breach: {tracker.current_rate():.1%}")# 99.5% availability SLO → 0.5% error budget = 43.8 hours of downtime/year
def compute_error_budget(target_availability: float, window_days: int = 30) -> dict:
error_budget_pct = 1 - target_availability
total_minutes = window_days * 24 * 60
return {
"error_budget_minutes": total_minutes * error_budget_pct,
"error_budget_hours": total_minutes * error_budget_pct / 60,
}
# Alert when burn rate is 10x the normal rate
def should_alert_on_burn_rate(
consumed_pct: float, # How much error budget consumed
time_elapsed_pct: float, # What fraction of window elapsed
multiplier: float = 10,
) -> bool:
burn_rate = consumed_pct / time_elapsed_pct if time_elapsed_pct > 0 else 0
return burn_rate > multiplierWhat makes a user say "this AI is broken"? Slow responses? Wrong answers? Format errors? Those are your SLI candidates.
Run your metric for 2 weeks to understand the baseline. A target of 99% sounds good until you discover your current rate is 85%.
Start at current performance + 5%. Stretch goals nobody hits are ignored. Achievable targets get tracked.
p95 < 5s is a user-facing SLO. Cache hit rate > 30% is an internal efficiency SLO. Don't put both in your customer-facing SLA.
Quality SLOs need sampling strategies.You can't score 100% of production traffic with an LLM judge — it doubles your API costs. Use a 5–10% sample rate, but ensure the sample is stratified: sample across all operation types and tenants, not just the highest-volume traffic.
A service level indicator (SLI) is a measured number, such as the fraction of requests that completed without a 5xx error. A service level objective (SLO) is the target you set for that number, such as 99.5% over 30 days. A service level agreement (SLA) is a contract with customers that defines consequences, often refunds, when a target is missed. Keep your internal SLO stricter than any SLA so that you notice trouble before you owe anyone anything.
An SLO of 99.5% means 0.5% of events may fail. Over a 30-day window that is 30 × 24 × 60 = 43,200 minutes, so the time-based budget is 43,200 × 0.005 = 216 minutes, or 3.6 hours. AI services are usually better measured per request than per minute: if you serve 1,000,000 requests in the window, you may fail 1,000,000 × 0.005 = 5,000 of them. That count is your error budget, and it is what you spend on deployments, provider incidents, and experiments.
Burn rate is how fast you are spending the budget relative to an even pace. Suppose 10% of the window has passed (3 days) and 40% of the budget is gone (2,000 failures). The burn rate is 40% / 10% = 4. At that pace the budget would be exhausted after 30 / 4 = 7.5 days. A burn rate of 1 means you would use exactly the whole budget at the end of the window.
def burn_rate(error_rate_now: float, slo: float) -> float:
"""How many times faster than 'sustainable' are we failing right now?"""
allowed = 1 - slo # 0.005 for a 99.5% SLO
return error_rate_now / allowed
print(burn_rate(0.005, 0.995)) # 1.0 exactly on budget
print(burn_rate(0.02, 0.995)) # 4.0 2% of requests failing
print(burn_rate(0.072, 0.995)) # 14.4 budget gone in 30 / 14.4 = about 2.1 days
# Share of a 30-day budget consumed in one hour at burn rate 14.4:
print(14.4 * 1 / (30 * 24)) # 0.02 -> 2% of the monthly budget in a single hourThis is why alerts are often tied to burn rate with two windows: page someone when a fast burn (for example, a burn rate around 14 sustained over an hour) is eating a meaningful slice of the budget, and open a ticket for a slow burn (a burn rate around 1 to 3 over a day or more) that will exhaust the budget later without being urgent today. Requiring both a short and a long window to exceed the threshold reduces false pages from brief spikes.
Quality SLOs depend on a sampled judge, so the measured rate has statistical noise. Suppose 1,000 requests a day with a 10% sample gives 100 judged responses a day, or 700 in a 7-day window. If the true share scoring at least 7/10 is 0.90, the standard error of the measured share is √(0.90 × 0.10 / 700) = √0.000129 = 0.0113, so the 7-day number typically wanders by about ±2.2 points (1.96 standard errors) from week to week with no real change. That is why the alert threshold in this lesson sits at 85% for a 90% target: the gap leaves room for noise. A daily window with only 100 samples has a standard error of √(0.09 / 100) = 0.03, about ±5.9 points, and would alert falsely far too often.
The same effect hurts low-traffic availability alerts. With 20 requests in a five-minute window, one failure is a 5% error rate, which looks like a catastrophic breach of a 99.5% SLO but is a single request. For low-volume services use longer windows or a minimum event count before alerting.
The p95 latency is the value below which 95% of requests fall. Two things to remember. You cannot average percentiles across servers or time windows: if one server has a p95 of 1 s and another 9 s, the combined p95 is not 5 s. Aggregate the raw samples or use histograms, then compute the percentile. And LLM latency is long-tailed because output length varies, so the mean can look healthy while p95 is poor. Consider separate SLOs for time to first token (what streaming users feel) and total completion time.
Choose indicators that reflect what users feel, set targets from measured baselines, convert them to error budgets in requests, and alert on burn rate over short and long windows. Account for sampling noise in quality metrics, never average percentiles, and use the budget to decide when to ship and when to stabilize.