Full fine-tuning a 7B model requires 80GB of GPU RAM. LoRA achieves 95% of that quality with 0.06% of the parameters — by freezing the base model and training small adapter matrices. QLoRA pushes it further: a 7B model on a consumer GPU.
Every attention weight matrix W in a transformer has dimensions d×d. LoRA adds two small matrices A (d×r) and B (r×d) where rank r ≪ d. Instead of updating W directly, we learn ΔW = BA. Since r is tiny (4–64 vs d=4096), trainable parameters collapse from d² to 2dr.
# LoRA math W_base = frozen_weights # d × d = 4096 × 4096 = 16.7M params (frozen) A = random_init(d, r) # 4096 × 8 = 32K params (trained) B = zeros(r, d) # 8 × 4096 = 32K params (trained) # Forward pass: h = (W_base + A @ B) @ x # = h_base + delta_h # Trainable params: 2 × d × r = 2 × 4096 × 8 = 65,536 # vs full fine-tuning: 4096 × 4096 = 16,777,216 # → 256× fewer trainable parameters! # After training: merge LoRA into base model for zero inference overhead W_merged = W_base + (B @ A) * scale # Now W_merged behaves like a fully fine-tuned weight, with no adapter overhead
Adds small adapter matrices to attention layers. The base model is frozen. Only the adapters are trained. At inference, adapters are merged — zero latency overhead.
# pip install transformers peft datasets trl bitsandbytes accelerate
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer, TrainingArguments
MODEL_NAME = "meta-llama/Llama-3.2-7B-Instruct"
# ─── Step 1: Load in 4-bit (QLoRA) ──────────────────────────
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True, # Nested quantization
bnb_4bit_quant_type="nf4", # NormalFloat4 — best for LLMs
bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
MODEL_NAME,
quantization_config=bnb_config,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
# ─── Step 2: Configure LoRA adapters ─────────────────────────
lora_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=16, # Rank — higher = more capacity, more params
lora_alpha=32, # Scaling factor (alpha/r = effective scale)
target_modules=[ # Which weight matrices to adapt
"q_proj", "k_proj", "v_proj", "o_proj", # Attention
"gate_proj", "up_proj", "down_proj", # MLP (optional)
],
lora_dropout=0.1,
bias="none",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# → trainable params: 3,407,872 || all params: 6,742,609,920 || trainable%: 0.05
# ─── Step 3: Train ───────────────────────────────────────────
trainer = SFTTrainer(
model=model,
args=TrainingArguments(
output_dir="./qlora-output",
num_train_epochs=3,
per_device_train_batch_size=2,
gradient_accumulation_steps=8, # Effective batch = 16
learning_rate=2e-4,
warmup_ratio=0.03,
fp16=True,
optim="paged_adamw_8bit", # Memory-efficient optimizer
),
train_dataset=dataset,
dataset_text_field="text",
max_seq_length=2048,
)
trainer.train()
# ─── Step 4: Merge and save (for deployment) ─────────────────
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, torch_dtype=torch.float16)
merged = PeftModel.from_pretrained(base_model, "./qlora-output")
merged = merged.merge_and_unload() # Bake LoRA into base weights
merged.save_pretrained("./merged-model")| Rank (r) | Use Case | Extra Params |
|---|---|---|
| r=4 | Simple style/format tasks | ~1M |
| r=8 | Most fine-tuning tasks (recommended default) | ~2M |
| r=16 | Complex domain adaptation | ~4M |
| r=64 | Near-full fine-tuning quality on hard tasks | ~16M |
With the training mechanics covered, the next lesson focuses on the piece most engineers underinvest in: preparing a high-quality fine-tuning dataset.