Changing a prompt is a code change. Like any code change, it should be validated before full rollout. A/B testing AI features lets you measure whether “version B” is actually better — with statistical confidence — before it reaches all users.
import hashlib
def assign_variant(user_id: str, experiment_id: str, control_ratio: float = 0.5) -> str:
"""
Deterministic assignment — same user always gets the same variant.
Hash (user_id + experiment_id) so different experiments don't correlate.
"""
key = f"{user_id}:{experiment_id}"
hash_val = int(hashlib.md5(key.encode()).hexdigest(), 16) / (16**32)
return "control" if hash_val < control_ratio else "treatment"
# Middleware — assign before the request reaches the LLM
def get_prompt(user_id: str) -> str:
variant = assign_variant(user_id, experiment_id="prompt-v2-test")
if variant == "control":
return PROMPT_V1 # Current production prompt
return PROMPT_V2 # New challenger prompt
# Log variant with every response
log_event({
"user_id": user_id,
"experiment_id": "prompt-v2-test",
"variant": variant,
"response": answer,
"latency_ms": latency_ms,
})Treatment leaks into control via shared cache. Fix: hash cache keys by variant, or disable caching during experiments.
Checking significance daily and stopping when p < 0.05. Fix: set sample size before the experiment; do not analyze until you reach it.
Optimizing for LLM judge score when users care about latency. Fix: instrument the metric users actually experience (thumbs up/down, task completion rate).
Users engage more with the new version simply because it's new. Fix: run experiments for ≥ 2 weeks, or exclude first interactions.
Minimum viable experiment:For most AI feature changes, you need 200–500 responses per variant to detect a 5% quality improvement at 80% statistical power. At 1,000 requests/day with 50% sampling, that's a 4–10 day experiment. Shorter tests produce unreliable conclusions.
Many AI features are judged by a binary signal such as thumbs-up rate. Suppose the current prompt earns a 60% thumbs-up rate and you want to detect an improvement to 65% with a 5% false-positive rate (two-sided) and 80% power. The standard sample-size formula for comparing two proportions is n = (zα/2 + zβ)² · [p₁(1−p₁) + p₂(1−p₂)] / (p₂ − p₁)² per variant.
z_alpha = 1.96 # two-sided alpha = 0.05
z_beta = 0.8416 # power = 0.80
p1, p2 = 0.60, 0.65
variance_sum = p1*(1-p1) + p2*(1-p2) # 0.24 + 0.2275 = 0.4675
n = (z_alpha + z_beta)**2 * variance_sum / (p2 - p1)**2
# (2.8016)^2 = 7.849; 7.849 * 0.4675 = 3.669; 3.669 / 0.0025 = 1467.7
print(round(n)) # 1468 users per variant, about 2,936 in totalThis is larger than the rule of thumb in the note below because binary outcomes are noisy: a 5-point gap on a 60% base needs well over a thousand users per arm. The needed size grows with the square of how small an effect you want to detect, so halving the target lift roughly quadruples the traffic.
Now read a result. Say control got 600 thumbs-up out of 1,000 (0.60) and treatment got 650 out of 1,000 (0.65). A two-proportion z-test pools the rates under the null hypothesis that both variants are the same:
import math
x1, n1 = 600, 1000
x2, n2 = 650, 1000
p1, p2 = x1/n1, x2/n2
pooled = (x1 + x2) / (n1 + n2) # 1250/2000 = 0.625
se = math.sqrt(pooled * (1 - pooled) * (1/n1 + 1/n2))
# sqrt(0.625 * 0.375 * 0.002) = sqrt(0.00046875) = 0.02165
z = (p2 - p1) / se # 0.05 / 0.02165 = 2.31
p_value = math.erfc(abs(z) / math.sqrt(2)) # two-sided, about 0.021
# 95% confidence interval for the difference (unpooled SE)
se_diff = math.sqrt(p1*(1-p1)/n1 + p2*(1-p2)/n2) # sqrt(0.0004675) = 0.02162
low, high = (p2 - p1) - 1.96*se_diff, (p2 - p1) + 1.96*se_diff
print(z, p_value, low, high) # 2.31, 0.021, 0.0076, 0.0924The p-value of about 0.021 is below 0.05, so the difference is statistically significant. Notice the interval, though: the true lift could be anywhere from under one point to over nine. Significance tells you the effect is probably not zero; the interval tells you whether it is big enough to justify the change. If the new prompt also costs 30% more tokens, a lift whose lower bound is 0.8 points may not be worth shipping. Decide the smallest lift that matters before the test starts.
A single quality metric is not enough. Track guardrail metrics that must not get worse: p95 latency, cost per request, refusal rate, error rate. A prompt that raises thumbs-up rate but doubles output length may be a net loss. Evaluate guardrails with their own tests, and treat a significant regression as a reason to stop even when the primary metric improved.
Also verify the split itself. If you intended 50/50 and 10,000 users arrived as 5,200 control and 4,800 treatment, a chi-square check gives (5,200−5,000)²/5,000 + (4,800−5,000)²/5,000 = 8 + 8 = 16, with one degree of freedom. That corresponds to a p-value near 0.00006, so the imbalance is not chance. This sample ratio mismatchusually means a bug: one variant crashes more often, a cache serves one arm, or bots are filtered unevenly. Fix the bug before trusting any metric from that experiment.
Decide the hypothesis and the smallest lift that matters, compute the sample size up front, assign users deterministically, check the split for mismatch, and read the confidence interval along with the p-value. Guard cost and latency with their own metrics, and resist stopping early. A disciplined small test beats a quick one that you cannot trust.
The final lesson pulls together everything from this course into a full system design capstone: cascade, cache, circuit breakers, load test, and A/B test — all wired into one production architecture.