Full fine-tuning a 7B model requires 80GB of GPU RAM. LoRA achieves 95% of that quality with 0.06% of the parameters — by freezing the base model and training small adapter matrices. QLoRA pushes it further: a 7B model on a consumer GPU.
Every attention weight matrix W in a transformer has dimensions d×d. LoRA adds two small matrices A (d×r) and B (r×d) where rank r ≪ d. Instead of updating W directly, we learn ΔW = BA. Since r is tiny (4–64 vs d=4096), trainable parameters collapse from d² to 2dr.
# LoRA math W_base = frozen_weights # d × d = 4096 × 4096 = 16.7M params (frozen) A = random_init(d, r) # 4096 × 8 = 32K params (trained) B = zeros(r, d) # 8 × 4096 = 32K params (trained) # Forward pass: h = (W_base + A @ B) @ x # = h_base + delta_h # Trainable params: 2 × d × r = 2 × 4096 × 8 = 65,536 # vs full fine-tuning: 4096 × 4096 = 16,777,216 # → 256× fewer trainable parameters! # After training: merge LoRA into base model for zero inference overhead W_merged = W_base + (B @ A) * scale # Now W_merged behaves like a fully fine-tuned weight, with no adapter overhead
Adds small adapter matrices to attention layers. The base model is frozen. Only the adapters are trained. At inference, adapters are merged — zero latency overhead.
# pip install transformers peft datasets trl bitsandbytes accelerate
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer, TrainingArguments
MODEL_NAME = "meta-llama/Llama-3.2-7B-Instruct"
# ─── Step 1: Load in 4-bit (QLoRA) ──────────────────────────
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True, # Nested quantization
bnb_4bit_quant_type="nf4", # NormalFloat4 — best for LLMs
bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
MODEL_NAME,
quantization_config=bnb_config,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
# ─── Step 2: Configure LoRA adapters ─────────────────────────
lora_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=16, # Rank — higher = more capacity, more params
lora_alpha=32, # Scaling factor (alpha/r = effective scale)
target_modules=[ # Which weight matrices to adapt
"q_proj", "k_proj", "v_proj", "o_proj", # Attention
"gate_proj", "up_proj", "down_proj", # MLP (optional)
],
lora_dropout=0.1,
bias="none",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# → trainable params: 3,407,872 || all params: 6,742,609,920 || trainable%: 0.05
# ─── Step 3: Train ───────────────────────────────────────────
trainer = SFTTrainer(
model=model,
args=TrainingArguments(
output_dir="./qlora-output",
num_train_epochs=3,
per_device_train_batch_size=2,
gradient_accumulation_steps=8, # Effective batch = 16
learning_rate=2e-4,
warmup_ratio=0.03,
fp16=True,
optim="paged_adamw_8bit", # Memory-efficient optimizer
),
train_dataset=dataset,
dataset_text_field="text",
max_seq_length=2048,
)
trainer.train()
# ─── Step 4: Merge and save (for deployment) ─────────────────
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, torch_dtype=torch.float16)
merged = PeftModel.from_pretrained(base_model, "./qlora-output")
merged = merged.merge_and_unload() # Bake LoRA into base weights
merged.save_pretrained("./merged-model")| Rank (r) | Use Case | Extra Params |
|---|---|---|
| r=4 | Simple style/format tasks | ~1M |
| r=8 | Most fine-tuning tasks (recommended default) | ~2M |
| r=16 | Complex domain adaptation | ~4M |
| r=64 | Near-full fine-tuning quality on hard tasks | ~16M |
In the PEFT library the learned update is ΔW = (lora_alpha / r) · B·A, where A has shape r×d_in and B has shape d_out×r. B starts at zero, so at step zero the adapted model behaves exactly like the base model and training departs smoothly from there. The rank r sets how many directions the update can use, which bounds its capacity. The ratio lora_alpha / r scales how strongly the update is applied: doubling alpha at a fixed rank has an effect similar to raising the learning rate for the adapter. A common starting point is lora_alpha equal to r or 2×r, then adjust only if training is unstable or underfits.
target_modules chooses which layers get adapters. Attention projections alone are cheaper; adding the MLP projections (gate_proj, up_proj, down_proj) usually gives the adapter more capacity at a modest memory cost. Module names differ between architectures, so print model and read the layer names before copying a list from another model.
Training memory has several parts: the frozen base weights, the adapter weights, gradients and optimizer state for the adapters, and activations. LoRA removes gradients and optimizer state for the base model, which is the largest saving. QLoRA additionally stores the frozen base weights in 4-bit NormalFloat (NF4), shrinking the weights to roughly a quarter of their 16-bit size. During the forward and backward pass the 4-bit blocks are dequantized on the fly to the compute dtype (bfloat16 above), while gradients flow only into the 16-bit adapters. The cost is some extra compute for dequantization, so QLoRA is usually slower per step than LoRA on the same hardware. Activations still grow with batch size and sequence length, which is why gradient accumulation and shorter max_seq_length are the first levers when you run out of memory.
The script above expects a dataset with a textcolumn. For instruction tuning, each row should be the full prompt-plus-answer string in the format the base model was trained on, usually produced by the tokenizer's chat template:
from datasets import Dataset
rows = [
{"question": "Reset my password?", "answer": "Go to Settings > Security > Reset password."},
# ... hundreds more, reviewed by a human
]
def to_text(row):
messages = [
{"role": "user", "content": row["question"]},
{"role": "assistant", "content": row["answer"]},
]
return {"text": tokenizer.apply_chat_template(messages, tokenize=False)}
dataset = Dataset.from_list(rows).map(to_text)
split = dataset.train_test_split(test_size=0.1, seed=42) # keep a held-out setHold out a validation split and watch its loss. If training loss keeps falling while validation loss rises, the adapter is memorizing the examples.
LoRA freezes the base model and learns a low-rank update, so only a tiny fraction of parameters train. QLoRA stores the frozen base in 4-bit to fit larger models in less memory. Rank and alpha control capacity and strength, target modules decide where adapters go, and data format and a held-out validation set decide whether the result is actually useful. Always compare against the untuned model before shipping.
With the training mechanics covered, the next lesson focuses on the piece most engineers underinvest in: preparing a high-quality fine-tuning dataset.