This capstone produces a production-ready domain expert model: choose a domain, collect data, train a QLoRA model, evaluate it against a prompted Claude baseline, deploy it, and document when fine-tuning was and was not worth it.
Pick one with clear right/wrong outputs — easier to evaluate:
Collect or generate 1,000 training examples
Use claude-haiku-4-5-20251001 for synthetic generation with 10 manual seed examples. Budget: ~$0.50
Create 100-example test set (held out)
Write these manually or use a different generation process — must not overlap with training data
Run deduplication (MinHash LSH) and quality filters
Log how many examples were removed and why. Target <5% removal rate.
Manual review of 50 random training examples
Document any systematic issues found. Fix before training.
Fine-tune Llama 3.2 3B with QLoRA (r=16)
Use training script from Lesson 5. Log to wandb. 3 epochs, lr=2e-4.
Monitor train and eval loss curves
Screenshot the wandb loss chart. Note the epoch where eval loss bottoms out.
Merge LoRA adapters and save merged model
merge_and_unload() — required for efficient inference
Run the catastrophic forgetting test battery
5 general-knowledge questions. Log base vs fine-tuned responses side by side.
import anthropic
import time
client = anthropic.Anthropic()
def claude_baseline(question: str, domain_context: str) -> str:
"""Zero-shot claude-sonnet-4-6 with a strong system prompt."""
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=500,
system=f"You are an expert in {domain_context}. Be precise and follow the exact output format requested.",
messages=[{"role": "user", "content": question}],
)
return response.content[0].text
# Compare on 100 test cases:
results = {
"finetuned": {"scores": [], "latency_ms": []},
"claude_sonnet": {"scores": [], "latency_ms": []},
}
for test_case in test_cases:
# Fine-tuned model
t0 = time.monotonic()
ft_output = finetuned_model.generate(test_case["input"])
results["finetuned"]["latency_ms"].append(int((time.monotonic() - t0) * 1000))
# Claude baseline
t0 = time.monotonic()
claude_output = claude_baseline(test_case["input"], domain)
results["claude_sonnet"]["latency_ms"].append(int((time.monotonic() - t0) * 1000))
# Score both
ft_score = evaluate(test_case, ft_output)
claude_score = evaluate(test_case, claude_output)
results["finetuned"]["scores"].append(ft_score)
results["claude_sonnet"]["scores"].append(claude_score)
# Report
for model_name, data in results.items():
print(f"{model_name}:")
print(f" Accuracy: {sum(data['scores'])/len(data['scores']):.1%}")
print(f" Avg latency: {sum(data['latency_ms'])//len(data['latency_ms'])}ms")Write a 1-page "Was fine-tuning worth it?" analysis
Compare: accuracy gain, cost to train, inference cost, latency. When would you recommend fine-tuning to a client?
Deploy fine-tuned model via Modal or HF Dedicated Endpoint
Include endpoint URL and curl example in your documentation
Push model card to Hugging Face Hub
Document base model, dataset size, eval results, intended use, limitations
Congratulations on completing LLM Fine-Tuning & Adaptation. You now know how to train, evaluate, and serve custom models. The final course, AI Agent Frameworks, teaches you to orchestrate multiple AI models into multi-step autonomous systems.