Training loss reaching 0.3 means nothing. The test that matters is: does your fine-tuned model outperform the base model on your specific task — without breaking everything else?
Select a training outcome to see what the loss curve looks like and how to respond:
Signal: Train loss decreases smoothly. Eval loss tracks train loss with a small gap, then plateaus.
Fix: Stop here — this is the target. Deploy the checkpoint just before eval loss starts rising.
# Healthy convergence — example loss values
epoch train_loss eval_loss delta
1 2.31 2.38 +0.07 ← train/eval tracking closely
2 1.84 1.95 +0.11
3 1.42 1.58 +0.16
4 1.12 1.29 +0.17
5 0.89 1.08 +0.19 ← plateau approaching
6 0.74 1.06 +0.32 ← eval loss stopped improving → STOP
7 0.61 1.11 +0.50 ← would be overfitting from here
# Save checkpoint from epoch 5 or 6
trainer.save_model("./best-checkpoint-epoch5")The training loss curve is your primary diagnostic. Four patterns to recognize:
Train loss decreases smoothly. Eval loss tracks train loss with a small gap. Eval loss plateaus and then slightly increases — stop here.
Train loss continues decreasing. Eval loss increases. Gap between them widens. Fix: reduce epochs, add dropout, or get more data.
Both train and eval loss plateau high (never below 1.5 for generation). Fix: increase learning rate, reduce warmup, or check data format.
Task accuracy improves but model loses general reasoning. Happens with too high LR or too many epochs. Fix: reduce LR to 1e-5, use fewer epochs.
from transformers import pipeline
import json
import anthropic
# Load fine-tuned vs base model for comparison
finetuned = pipeline("text-generation", model="./merged-model", device_map="auto")
base = pipeline("text-generation", model="meta-llama/Llama-3.2-3B-Instruct", device_map="auto")
client = anthropic.Anthropic()
def score_response(task: str, response: str, expected: str) -> float:
"""Use LLM judge to score a response 0.0–1.0."""
judge = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=50,
messages=[{
"role": "user",
"content": f"""Score this response from 0 to 10.
Task: {task}
Expected: {expected}
Got: {response}
Output only the number.""",
}],
)
return float(judge.content[0].text.strip()) / 10
# Run head-to-head comparison
test_cases = json.loads(open("data/test.json").read())
ft_scores, base_scores = [], []
for case in test_cases:
prompt = f"### Instruction:\n{case['instruction']}\n\n### Input:\n{case['input']}\n\n### Response:\n"
ft_output = finetuned(prompt, max_new_tokens=200, do_sample=False)[0]["generated_text"][len(prompt):]
base_output = base(prompt, max_new_tokens=200, do_sample=False)[0]["generated_text"][len(prompt):]
ft_score = score_response(case["instruction"], ft_output, case["output"])
base_score = score_response(case["instruction"], base_output, case["output"])
ft_scores.append(ft_score)
base_scores.append(base_score)
print(f"Fine-tuned avg: {sum(ft_scores)/len(ft_scores):.2f}")
print(f"Base model avg: {sum(base_scores)/len(base_scores):.2f}")
print(f"Improvement: {(sum(ft_scores)-sum(base_scores))/len(ft_scores):+.2f}")Fine-tuning on a narrow task can degrade general capabilities. Always run a sanity check:
FORGETTING_TESTS = [
{
"prompt": "What is the capital of Japan?",
"expected": "Tokyo",
},
{
"prompt": "Write a Python function to reverse a string.",
"expected": "def reverse_string(s): return s[::-1]",
},
{
"prompt": "Explain the difference between TCP and UDP.",
"expected": "TCP is connection-oriented with guaranteed delivery; UDP is connectionless and faster.",
},
]
print("\n=== Catastrophic Forgetting Check ===")
for test in FORGETTING_TESTS:
ft_resp = finetuned(test["prompt"], max_new_tokens=100, do_sample=False)[0]["generated_text"]
base_resp = base(test["prompt"], max_new_tokens=100, do_sample=False)[0]["generated_text"]
ft_ok = test["expected"].lower() in ft_resp.lower()
base_ok = test["expected"].lower() in base_resp.lower()
status = "✓" if ft_ok else "✗ FORGOTTEN"
print(f"{status} | {test['prompt'][:50]}")| Signal | Action |
|---|---|
| Task accuracy > base by >15% | Ship — fine-tuning succeeded |
| Task accuracy > base by 5–15% | Acceptable — collect more data for next iteration |
| Task accuracy ≤ base | Do not ship — revert. Check data quality, LR, epoch count |
| General capability drops >10% | Reduce epochs or LR — catastrophic forgetting is occurring |
| JSON parse error rate >10% | Add more format-consistency examples to training set |
The iteration loop. Fine-tuning is iterative. Your first run will almost never be the best. Typical cycle: train → evaluate → identify failure patterns → fix data → retrain. Budget for 3–5 iterations before production deployment.
Next: you have a fine-tuned model. How do you serve it in production without paying $5/hour for a dedicated A100?