Aligning AI: An Introduction to RLHF and DPO | FromZero2AI Blog

After an LLM completes its pre-training phase on large corpora of text, it becomes excellent at predicting the next token. However, a pre-trained base model is not necessarily a helpful assistant; it may complete prompts by repeating them, generating low-quality content, or outputting toxic or unsafe language.

To transform base models into helpful, honest, and harmless conversational agents, researchers perform alignment. Two primary paradigms govern this process: Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO). In this article, we will break down the mechanics and differences between these two methodologies.

The Baseline: Supervised Fine-Tuning (SFT)

Before applying preference alignment, models undergo Supervised Fine-Tuning (SFT). During SFT, the model is trained on curated datasets of instruction-response pairs (e.g., "Write a python function..." followed by high-quality code). SFT teaches the model the format of instructions and answers, but it fails to address nuanced human preferences or prevent subtle, undesirable behaviors.

Reinforcement Learning from Human Feedback (RLHF)

RLHF has been the standard alignment framework for state-of-the-art models like GPT-4 and Claude. It is a multi-stage process that leverages reinforcement learning to optimize the model’s outputs.

Aligning AI: RLHF and DPO alignment pipeline flowchart Aligning AI: RLHF and DPO alignment pipeline flowchart

1. Training the Reward Model

First, human annotators are presented with a prompt x and multiple model-generated responses (e.g., y_1, y_2). Annotators rank these responses from best to worst, creating preference pairs where y_w is the winning (preferred) response and y_l is the losing (dispreferred) response.

A Reward Model R_\phi is trained on this data. It outputs a scalar score representing how much humans would favor a given response. The loss function for training the reward model is:

\mathcal{L}_{\text{RM}}(\phi) = - \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( R_\phi(x, y_w) - R_\phi(x, y_l) \right) \right]

Where \sigma is the sigmoid function. This maximizes the score difference between the winning and losing completions.

2. Reinforcement Learning via PPO

Once the reward model is trained, the SFT model is updated using Proximal Policy Optimization (PPO). The agent is the language model policy \pi_\theta, and the state space is the sequence of tokens. The policy generates a completion, the reward model scores it, and PPO updates the policy weights to maximize the expected reward.

To prevent the model from drifting too far from the original SFT distribution and generating gibberish (a phenomenon called reward hacking), a Kullback-Leibler (KL) divergence penalty is added to the objective:

\text{Objective}(\theta) = \mathbb{E}_{(x, y) \sim \pi_\theta} \left[ R_\phi(x, y) \right] - \beta \, \mathbb{D}_{\text{KL}} \left( \pi_\theta(y | x) \,\|\, \pi_{\text{ref}}(y | x) \right)

Where \pi_{\text{ref}} is the frozen SFT model, and \beta controls the strength of the KL penalty.

Challenges of RLHF

RLHF is notoriously unstable. It requires running four large models in memory simultaneously during training: the active policy \pi_\theta, the reference model \pi_{\text{ref}}, the reward model R_\phi, and a value model used by PPO. This makes it resource-heavy and sensitive to hyperparameters.

Direct Preference Optimization (DPO)

Developed by researchers at Stanford, Direct Preference Optimization (DPO) simplifies alignment by eliminating the need for a separate reward model and reinforcement learning step.

DPO mathematically proves that the RLHF objective can be optimized directly using a simple binary cross-entropy loss over preference pairs. The key insight is that the implicit reward function can be expressed directly in terms of the policy \pi_\theta and reference policy \pi_{\text{ref}}:

R^*(x, y) = \beta \log \frac{\pi_\theta(y | x)}{\pi_{\text{ref}}(y | x)}

Substituting this back into the preference loss yields the DPO loss function:

\mathcal{L}_{\text{DPO}}(\theta) = - \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w | x)}{\pi_{\text{ref}}(y_w | x)} - \beta \log \frac{\pi_\theta(y_l | x)}{\pi_{\text{ref}}(y_l | x)} \right) \right]

How DPO Works Intuitively

The loss function forces the active policy \pi_\theta to increase the likelihood of the preferred response y_w relative to the reference policy \pi_{\text{ref}}, while simultaneously decreasing the likelihood of the dispreferred response y_l. This achieves the same mathematical optimum as RLHF but with single-stage supervised training.

Implementing DPO with the TRL Library

Below is a Python implementation utilizing the Hugging Face trl (Transformer Reinforcement Learning) library to train a model using DPO.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer, DPOConfig
from datasets import load_dataset

# 1. Load Tokenizer and Models
model_id = "Qwen/Qwen2.5-0.5B-Instruct"

model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=torch.bfloat16, 
    device_map="auto"
)
# DPO requires a reference model. If left None, DPOTrainer will clone 
# the training model and freeze it to act as the reference.
ref_model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=torch.bfloat16, 
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

# 2. Load and Format Dataset
# DPO datasets need columns: 'prompt', 'chosen', and 'rejected'
dataset = load_dataset("Intel/orca_dpo_pairs", split="train[:1000]")

# 3. Configure DPO Trainer Parameters
training_args = DPOConfig(
    output_dir="./dpo_results",
    beta=0.1,  # Temperature hyperparameter for DPO (implicit reward scale)
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    learning_rate=5e-6,
    logging_steps=10,
    max_length=512,
    max_prompt_length=256,
    remove_unused_columns=False,
    fp16=False,
    bf16=True
)

# 4. Initialize and Run Trainer
dpo_trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
)

dpo_trainer.train()

Comparison: RLHF vs. DPO

Choosing Between Them

The table above makes DPO look strictly better, and for most teams it is — but the comparison hides a real asymmetry. RLHF is on-policy: the reward model scores completions the current policy actually generates, so it keeps supervising the model as its behaviour shifts during training. DPO is off-policy. It learns from a fixed dataset of preference pairs produced by some other model, and once the policy drifts away from the distribution those pairs came from, the gradient signal is describing a model that no longer exists. This is why DPO runs frequently show a characteristic pathology: both the chosen and rejected log-probabilities fall over training. The loss only cares about the margin between them, and shrinking the rejected term faster than the chosen one satisfies it perfectly well while degrading the model's fluency overall.

Practically, that means watching the right metrics. Log chosen_rewards and rejected_rewards separately rather than tracking loss alone, and stop training when chosen rewards start trending down even if the margin is still widening. Keep beta in the 0.1-0.5 range and treat a low beta as what it is — permission for the policy to wander far from the reference. DPO also overfits fast; a single epoch over a preference set is often enough, and two is frequently worse than one.

The bigger caveat sits underneath both methods. Neither RLHF nor DPO can produce a model better than the preference data it was given. Annotator biases transfer directly — human raters demonstrably prefer longer, more confident, more agreeable answers, and models aligned on their judgements reliably become verbose and sycophantic regardless of which optimisation algorithm you used. If your aligned model has an obvious personality flaw, the algorithm is almost never the thing to change. Go re-read a hundred rows of your preference data instead; the flaw is usually sitting right there.