Lesson 7: A/B Testing AI Features

Changing a prompt is a code change. Like any code change, it should be validated before full rollout. A/B testing AI features lets you measure whether “version B” is actually better — with statistical confidence — before it reaches all users.

The AI A/B Testing Workflow

1. Define hypothesis: “PROMPT_V2 improves quality score by ≥5%”

2. Implement traffic assignment (50/50 split)

3. Log variant + quality metrics for every request

4. Collect minimum sample (run power analysis)

5. Run statistical test → ship, revert, or extend

import hashlib

def assign_variant(user_id: str, experiment_id: str, control_ratio: float = 0.5) -> str:
    """
    Deterministic assignment — same user always gets the same variant.
    Hash (user_id + experiment_id) so different experiments don't correlate.
    """
    key = f"{user_id}:{experiment_id}"
    hash_val = int(hashlib.md5(key.encode()).hexdigest(), 16) / (16**32)
    return "control" if hash_val < control_ratio else "treatment"


# Middleware — assign before the request reaches the LLM
def get_prompt(user_id: str) -> str:
    variant = assign_variant(user_id, experiment_id="prompt-v2-test")
    if variant == "control":
        return PROMPT_V1   # Current production prompt
    return PROMPT_V2       # New challenger prompt

# Log variant with every response
log_event({
    "user_id": user_id,
    "experiment_id": "prompt-v2-test",
    "variant": variant,
    "response": answer,
    "latency_ms": latency_ms,
})

Common A/B Testing Pitfalls in AI

SUTVA violation

Treatment leaks into control via shared cache. Fix: hash cache keys by variant, or disable caching during experiments.

Peeking (stopping early)

Checking significance daily and stopping when p < 0.05. Fix: set sample size before the experiment; do not analyze until you reach it.

Wrong metric

Optimizing for LLM judge score when users care about latency. Fix: instrument the metric users actually experience (thumbs up/down, task completion rate).

Novelty effect

Users engage more with the new version simply because it's new. Fix: run experiments for ≥ 2 weeks, or exclude first interactions.

Minimum viable experiment:For most AI feature changes, you need 200–500 responses per variant to detect a 5% quality improvement at 80% statistical power. At 1,000 requests/day with 50% sampling, that's a 4–10 day experiment. Shorter tests produce unreliable conclusions.

The final lesson pulls together everything from this course into a full system design capstone: cascade, cache, circuit breakers, load test, and A/B test — all wired into one production architecture.

import hashlib def assign_variant(user_id: str, experiment_id: str, control_ratio: float = 0.5) -> str: """ Deterministic assignment — same user always gets the same variant. Hash (user_id + experiment_id) so different experiments don't correlate. """ key = f"{user_id}:{experiment_id}" hash_val = int(hashlib.md5(key.encode()).hexdigest(), 16) / (16**32) return "control" if hash_val < control_ratio else "treatment" # Middleware — assign before the request reaches the LLM def get_prompt(user_id: str) -> str: variant = assign_variant(user_id, experiment_id="prompt-v2-test") if variant == "control": return PROMPT_V1 # Current production prompt return PROMPT_V2 # New challenger prompt # Log variant with every response log_event({ "user_id": user_id, "experiment_id": "prompt-v2-test", "variant": variant, "response": answer, "latency_ms": latency_ms, })