Supervised Fine-Tuning (SFT) is the same training algorithm as pre-training — next-token prediction — applied to a small, curated dataset. The mechanics are identical; the goal and data are different.
# SFT data format (Alpaca-style)
{
"instruction": "Extract the company name and founding year from this text.",
"input": "OpenAI was founded in 2015 by a group including Elon Musk.",
"output": "Company: OpenAI\nFounded: 2015"
}
# During SFT training, loss is computed ONLY on the output tokens
# (not on the instruction/input tokens — they are masked)
# This teaches: "given this instruction, generate this response"# pip install transformers datasets accelerate trl
# Requires: GPU with ≥16GB VRAM (A100/H100) or use QLoRA on 8GB
from datasets import Dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from trl import SFTTrainer
MODEL_NAME = "meta-llama/Llama-3.2-3B-Instruct" # 3B params — fits on 1x A100
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto")
# Format dataset into instruction–response pairs
def format_example(example: dict) -> dict:
text = (
f"<|system|>\nYou are a helpful assistant.\n"
f"<|user|>\n{example['instruction']}\n"
f"<|assistant|>\n{example['output']}"
)
return {"text": text}
raw_data = [
{"instruction": "Classify the sentiment of this review.", "output": "Positive"},
{"instruction": "Summarize this in one sentence.", "output": "The paper introduces a new method."},
# ... 1000+ more examples
]
dataset = Dataset.from_list(raw_data).map(format_example)
training_args = TrainingArguments(
output_dir="./finetuned-model",
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4, # Effective batch size = 16
warmup_steps=100,
learning_rate=2e-5,
fp16=True, # Mixed precision — reduces VRAM
logging_steps=50,
save_steps=500,
evaluation_strategy="steps",
eval_steps=500,
)
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset,
dataset_text_field="text",
max_seq_length=2048,
)
trainer.train()
trainer.save_model("./finetuned-llama-3b")| Hyperparameter | Typical Range | Effect |
|---|---|---|
| learning_rate | 1e-5 to 5e-5 | Too high → catastrophic forgetting. Too low → no learning. |
| num_train_epochs | 1–5 | More epochs → overfit to training set, lose generalization |
| per_device_batch_size | 1–16 (VRAM limited) | Larger = more stable gradients, more VRAM |
| warmup_steps | 50–200 | Prevents large gradient updates early in training |
| max_seq_length | 512–4096 | Match to your use case — longer costs more memory |
Training uses cross-entropy: for each target token, the loss is −ln(p), where p is the probability the model assigned to the correct token. Suppose the response part of an example is five tokens and the model currently assigns them probabilities 0.5, 0.25, 0.8, 0.1, and 0.9. The per-token losses are:
import math
probs = [0.5, 0.25, 0.8, 0.1, 0.9]
losses = [-math.log(p) for p in probs]
# [0.693, 1.386, 0.223, 2.303, 0.105] sum = 4.7105
mean_loss = sum(losses) / len(losses) # 4.7105 / 5 = 0.942
perplexity = math.exp(mean_loss) # e^0.942 = 2.57A perplexity of about 2.6 means the model is, on average, as uncertain as if it were choosing among roughly 2.6 equally likely tokens. The token with probability 0.1 contributes the most loss (2.303), which is how training focuses effort on what the model gets wrong. Training pushes these probabilities up for the responses in your dataset.
The instruction tokens are masked, so they contribute nothing. In Hugging Face-style code that means setting their label to -100, a value the loss function ignores:
prompt_ids = tokenizer("<|user|>\nClassify the review.\n<|assistant|>\n")["input_ids"]
response_ids = tokenizer("Positive")["input_ids"]
input_ids = prompt_ids + response_ids
labels = [-100] * len(prompt_ids) + response_ids # loss only on the responseWithout masking, the model also spends capacity learning to predict your instructions, which is wasted effort and can bias it toward copying prompt phrasing. Some trainer classes apply masking for you through a data collator; check what your version does rather than assuming.
Take the script above with 1,000 examples. The effective batch size is 4 per device × 4 accumulation steps × 1 device = 16. One epoch is ⌈1000 / 16⌉ = 63 optimizer steps, so three epochs is about 189 steps. That has two consequences. The script sets warmup_steps=100, which would spend more than half of the entire run warming up, and save_steps=500 and eval_steps=500 would never trigger. For a dataset this small, use something like 10 warmup steps and evaluate every 20 to 50 steps. Always do this arithmetic for your own dataset before launching a run.
Memory is the other planning number. Full fine-tuning with the Adam optimizer in mixed precision needs, per parameter, about 2 bytes for the fp16 weights, 2 for gradients, 4 for an fp32 master copy, and 8 for the two Adam moments, roughly 16 bytes in total. For a 3-billion-parameter model that is 3 × 10⁹ × 16 B ≈ 48 GB before counting activations, which is why the comment in the script mentions larger GPUs and why parameter-efficient methods like LoRA are the usual choice on smaller hardware. This is a standard rough estimate, not a measured figure.
SFT is good at teaching format, tone, and task behavior: always answer in a given JSON schema, follow a house style, or classify tickets into your categories. It is a poor way to add large amounts of new factual knowledge. A few thousand examples may make the model repeat facts without reliably generalizing them, and may encourage confident errors on related questions. For changing knowledge, retrieval is usually the better tool; for stable behavior, SFT is.
Data quality matters more than quantity. A few hundred consistent, correct, diverse examples often beat thousands of noisy ones. Check that the examples cover the real distribution of inputs, include edge cases and refusals you want, and are deduplicated. Hold out a validation split the model never trains on.
tokenizer.apply_chat_template.eval_dataset. Without held-out data you cannot see overfitting.evaluation_strategy). Check the documentation for the version you install.SFT minimizes next-token cross-entropy on the response part of curated examples. Compute your steps, warmup, and memory before launching, mask the prompt, use the correct chat template, keep a validation split, and judge success against a fixed test set and a prompting baseline.
For most production fine-tuning, you won't train all model parameters — you'll use LoRA. The next lesson explains how LoRA achieves the same results with 95% fewer trainable parameters.