Supervised Fine-Tuning (SFT) is the same training algorithm as pre-training — next-token prediction — applied to a small, curated dataset. The mechanics are identical; the goal and data are different.
# SFT data format (Alpaca-style)
{
"instruction": "Extract the company name and founding year from this text.",
"input": "OpenAI was founded in 2015 by a group including Elon Musk.",
"output": "Company: OpenAI\nFounded: 2015"
}
# During SFT training, loss is computed ONLY on the output tokens
# (not on the instruction/input tokens — they are masked)
# This teaches: "given this instruction, generate this response"# pip install transformers datasets accelerate trl
# Requires: GPU with ≥16GB VRAM (A100/H100) or use QLoRA on 8GB
from datasets import Dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from trl import SFTTrainer
MODEL_NAME = "meta-llama/Llama-3.2-3B-Instruct" # 3B params — fits on 1x A100
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto")
# Format dataset into instruction–response pairs
def format_example(example: dict) -> dict:
text = (
f"<|system|>\nYou are a helpful assistant.\n"
f"<|user|>\n{example['instruction']}\n"
f"<|assistant|>\n{example['output']}"
)
return {"text": text}
raw_data = [
{"instruction": "Classify the sentiment of this review.", "output": "Positive"},
{"instruction": "Summarize this in one sentence.", "output": "The paper introduces a new method."},
# ... 1000+ more examples
]
dataset = Dataset.from_list(raw_data).map(format_example)
training_args = TrainingArguments(
output_dir="./finetuned-model",
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4, # Effective batch size = 16
warmup_steps=100,
learning_rate=2e-5,
fp16=True, # Mixed precision — reduces VRAM
logging_steps=50,
save_steps=500,
evaluation_strategy="steps",
eval_steps=500,
)
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset,
dataset_text_field="text",
max_seq_length=2048,
)
trainer.train()
trainer.save_model("./finetuned-llama-3b")| Hyperparameter | Typical Range | Effect |
|---|---|---|
| learning_rate | 1e-5 to 5e-5 | Too high → catastrophic forgetting. Too low → no learning. |
| num_train_epochs | 1–5 | More epochs → overfit to training set, lose generalization |
| per_device_batch_size | 1–16 (VRAM limited) | Larger = more stable gradients, more VRAM |
| warmup_steps | 50–200 | Prevents large gradient updates early in training |
| max_seq_length | 512–4096 | Match to your use case — longer costs more memory |
For most production fine-tuning, you won't train all model parameters — you'll use LoRA. The next lesson explains how LoRA achieves the same results with 95% fewer trainable parameters.