DPO & Modern Preference Optimization

1. The DPO Derivation

Direct Preference Optimization (Rafailov et al., 2023) begins with the same objective as RLHF: maximize expected reward while staying close to a reference policy. The standard RLHF objective is:

max_π E_{x~D, y~π}[r(x, y)] − β KL(π || π_ref)

The optimal solution to this constrained optimization problem has a closed-form expression:

π*(y|x) = π_ref(y|x) · exp(r(x, y) / β) / Z(x)

where Z(x) is the partition function that normalizes the distribution. The crucial step in DPO is rearranging this expression to solve for the reward in terms of the policy:

r(x, y) = β log(π(y|x) / π_ref(y|x)) + β log Z(x)

When we substitute this into the Bradley-Terry preference model, the partition function Z(x) cancels (since it appears in both the chosen and rejected terms), yielding the DPO loss:

L_DPO = −E[log σ(β(log π(y_w|x)/π_ref(y_w|x) − log π(y_l|x)/π_ref(y_l|x)))]

# DPO Training with TRL
from trl import DPOTrainer, DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

# Load SFT model as starting point
model = AutoModelForCausalLM.from_pretrained(
    "./sft-llama-8b-final",
    torch_dtype="bfloat16",
)
ref_model = AutoModelForCausalLM.from_pretrained(
    "./sft-llama-8b-final",
    torch_dtype="bfloat16",
)
tokenizer = AutoTokenizer.from_pretrained("./sft-llama-8b-final")

# Load preference dataset
# Must have: prompt, chosen, rejected columns
dataset = load_dataset("argilla/ultrafeedback-binarized-preferences", split="train")

print(f"Dataset size: {len(dataset)}")
print(f"Columns: {dataset.column_names}")
# prompt: the user query
# chosen: the preferred response
# rejected: the less-preferred response

# DPO training configuration
dpo_config = DPOConfig(
    output_dir="./dpo-llama-8b",
    beta=0.1,                    # KL penalty strength
    per_device_train_batch_size=2,
    gradient_accumulation_steps=16,
    learning_rate=5e-7,          # small LR for stability
    num_train_epochs=1,
    max_length=2048,
    max_prompt_length=1024,
    warmup_ratio=0.1,
    logging_steps=10,
    bf16=True,
    loss_type="sigmoid",         # standard DPO loss
    # Advanced options
    label_smoothing=0.0,         # 0.1 can help with noisy preferences
    precompute_ref_log_probs=True,  # saves memory
)

trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,
    args=dpo_config,
    train_dataset=dataset,
    tokenizer=tokenizer,
)

trainer.train()
trainer.save_model("./dpo-llama-8b-final")

2. DPO Variants and Extensions

The success of DPO inspired a wave of variants, each addressing specific limitations. The core differences lie in data requirements, loss formulations, and training dynamics.

2.1 KTO: Kahneman-Tversky Optimization

KTO (Ethayarajh et al., 2024) addresses a practical limitation of DPO: the requirement for paired preferences. In real applications, feedback often comes as binary signals (thumbs up or thumbs down) rather than pairwise comparisons. KTO works with unpaired binary feedback, using ideas from prospect theory to weight losses and gains asymmetrically.

# KTO Training with TRL
from trl import KTOTrainer, KTOConfig

# KTO uses unpaired binary data
# Each example has: prompt, completion, label (True/False)
kto_dataset = load_dataset("trl-lib/kto-mix-14k", split="train")

print(f"Example: {kto_dataset[0]}")
# {'prompt': '...', 'completion': '...', 'label': True}

kto_config = KTOConfig(
    output_dir="./kto-llama-8b",
    beta=0.1,
    desirable_weight=1.0,        # weight for positive examples
    undesirable_weight=1.0,      # weight for negative examples
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=5e-7,
    num_train_epochs=1,
    max_length=2048,
    bf16=True,
)

kto_trainer = KTOTrainer(
    model=model,
    ref_model=ref_model,
    args=kto_config,
    train_dataset=kto_dataset,
    tokenizer=tokenizer,
)

kto_trainer.train()

2.2 ORPO: Odds Ratio Preference Optimization

ORPO (Hong et al., 2024) eliminates the need for a separate reference model entirely. It combines the SFT objective with a preference optimization term in a single loss function. The key idea is to use the odds ratio of generating the chosen versus rejected response, contrasting them directly without a reference model baseline.

2.3 SimPO: Simple Preference Optimization

SimPO (Meng et al., 2024) also removes the reference model but takes a different approach. Instead of using log-probability ratios, SimPO uses the average log-probability of the response (normalized by length) as the implicit reward. It adds a target margin γ to the objective, encouraging a minimum quality gap between preferred and rejected responses.

2.4 IPO: Identity Preference Optimization

IPO (Azar et al., 2024) addresses a theoretical issue with DPO: under certain conditions, DPO can overfit to preference data, driving the log-probability ratio to infinity. IPO uses a squared loss instead of the sigmoid loss, providing better regularization properties and more stable training.

3. Creating Preference Datasets

The quality of alignment training depends critically on the preference dataset. Creating high-quality preference data involves careful annotation design, quality control, and understanding of common pitfalls.

3.1 Annotation Best Practices

• Clear guidelines: Define specific criteria for what makes a response "better" (accuracy, helpfulness, safety, conciseness)
• Multiple annotators: Use at least 2-3 annotators per comparison to measure agreement
• Calibration: Include known-answer items to detect annotator drift
• Diversity: Ensure prompts span different tasks, difficulty levels, and domains
• Margin filtering: Remove pairs where responses are nearly identical in quality (low signal-to-noise)

4. Synthetic Preference Generation

Human annotation is expensive and slow. A growing trend is to generate synthetic preference data using a stronger model (such as GPT-4 or Claude) as the judge. This approach, sometimes called "AI feedback" or RLAIF (Section 16.3), can produce large preference datasets at a fraction of the cost of human annotation.

# Synthetic preference generation with LLM-as-judge
import openai
from dataclasses import dataclass
from typing import List, Tuple

@dataclass
class PreferencePair:
    prompt: str
    chosen: str
    rejected: str
    judge_rationale: str

def generate_preference_pair(
    prompt: str,
    response_a: str,
    response_b: str,
    judge_model: str = "gpt-4o",
) -> PreferencePair:
    """Use a strong model to judge which response is better."""

    judge_prompt = f"""Compare these two responses to the given prompt.
Evaluate on: accuracy, helpfulness, clarity, and safety.
Return JSON with "winner" (A or B) and "rationale".

Prompt: {prompt}

Response A: {response_a}

Response B: {response_b}"""

    client = openai.OpenAI()
    result = client.chat.completions.create(
        model=judge_model,
        messages=[{"role": "user", "content": judge_prompt}],
        response_format={"type": "json_object"},
        temperature=0.0,
    )

    judgment = eval(result.choices[0].message.content)

    if judgment["winner"] == "A":
        return PreferencePair(prompt, response_a, response_b, judgment["rationale"])
    else:
        return PreferencePair(prompt, response_b, response_a, judgment["rationale"])


def build_synthetic_dataset(
    prompts: List[str],
    model_name: str = "meta-llama/Llama-3.1-8B-Instruct",
    samples_per_prompt: int = 4,
) -> List[PreferencePair]:
    """Build a preference dataset using rejection sampling + LLM judge."""
    import itertools

    pairs = []
    for prompt in prompts:
        # Generate multiple responses with different temperatures
        responses = []
        for temp in [0.3, 0.5, 0.7, 1.0]:
            response = generate_response(model_name, prompt, temperature=temp)
            responses.append(response)

        # Create all pairwise comparisons
        for a, b in itertools.combinations(responses, 2):
            pair = generate_preference_pair(prompt, a, b)
            pairs.append(pair)

    return pairs

5. Practical Considerations for DPO Training

5.1 Hyperparameter Sensitivity

DPO training is sensitive to several key hyperparameters. The most important is β, which controls the strength of the implicit KL constraint. A β that is too low leads to aggressive optimization that can degrade coherence. A β that is too high produces minimal change from the SFT model.

# Hyperparameter sweep for DPO
from dataclasses import dataclass
from typing import List, Dict

@dataclass
class DPOSweepConfig:
    """Configuration for DPO hyperparameter search."""
    beta_values: List[float] = None
    learning_rates: List[float] = None
    warmup_ratios: List[float] = None

    def __post_init__(self):
        self.beta_values = self.beta_values or [0.05, 0.1, 0.2, 0.5]
        self.learning_rates = self.learning_rates or [1e-7, 5e-7, 1e-6]
        self.warmup_ratios = self.warmup_ratios or [0.05, 0.1]

def evaluate_dpo_run(
    model_path: str,
    eval_dataset,
    metrics: List[str] = None,
) -> Dict[str, float]:
    """Evaluate a DPO checkpoint on standard metrics."""
    metrics = metrics or ["win_rate", "coherence", "kl_divergence"]
    results = {}

    # Win rate: how often the model's output is preferred
    # over the SFT baseline by an LLM judge
    results["win_rate"] = compute_win_rate(model_path, eval_dataset)

    # Coherence: perplexity on held-out text
    results["coherence"] = compute_perplexity(model_path, eval_dataset)

    # KL divergence from reference
    results["kl_divergence"] = compute_kl(model_path, eval_dataset)

    # Reward accuracy: agreement with held-out preferences
    results["reward_accuracy"] = compute_reward_accuracy(
        model_path, eval_dataset
    )

    return results

# Typical ranges for well-performing DPO
recommended_ranges = {
    "beta": "0.1 to 0.5 (start with 0.1)",
    "learning_rate": "1e-7 to 5e-6 (much lower than SFT)",
    "epochs": "1 to 3 (more can overfit)",
    "batch_size": "32 to 128 (larger is more stable)",
    "warmup_ratio": "0.05 to 0.15",
    "label_smoothing": "0.0 to 0.1 (helps with noisy data)",
}