Advanced PEFT Methods

1. DoRA: Weight-Decomposed Low-Rank Adaptation

DoRA (2024) improves on LoRA by decomposing the pretrained weight into its magnitude and direction components before applying the low-rank update. Specifically, a weight vector w is decomposed as w = m · (w / ||w||), where m is a learnable magnitude scalar and the direction is updated via standard LoRA. This decomposition aligns more closely with how full fine-tuning actually modifies weights, resulting in better performance for the same rank.

In practice, DoRA consistently outperforms LoRA by 1-3% across benchmarks when using the same rank and target modules, with only a marginal increase in trainable parameters (the additional magnitude vectors are tiny). The training speed is nearly identical to LoRA.

from peft import LoraConfig, get_peft_model

# DoRA configuration: simply enable use_dora flag
dora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_dropout=0.05,
    use_dora=True,          # Enable DoRA
    task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, dora_config)
model.print_trainable_parameters()
# Slightly more params than LoRA due to magnitude vectors,
# but typically ~1-3% better accuracy at same rank.

2. LoRA+: Asymmetric Learning Rates

LoRA+ (2024) addresses a subtle inefficiency in standard LoRA training. In the decomposition W' = W + BA, matrices B and A play different roles: A projects inputs into the low-rank space, while B projects back to the full space. LoRA+ assigns a higher learning rate to matrix B (typically 2x to 16x higher) than to matrix A, based on the theoretical analysis that the optimal learning rate ratio should scale with the model width.

In practice, LoRA+ improves convergence speed by 1.5x to 2x compared to standard LoRA, often reaching the same final quality in fewer steps. This is particularly valuable when training compute is the bottleneck.

from peft import LoraConfig, get_peft_model
from transformers import TrainingArguments
import torch

# LoRA+ uses different learning rates for A and B matrices
# This is implemented via parameter groups in the optimizer
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, lora_config)

# Separate parameter groups for A and B matrices
lora_A_params = []
lora_B_params = []
other_params = []

for name, param in model.named_parameters():
    if not param.requires_grad:
        continue
    if "lora_A" in name:
        lora_A_params.append(param)
    elif "lora_B" in name:
        lora_B_params.append(param)
    else:
        other_params.append(param)

# LoRA+ recommendation: lr_B = lr_ratio * lr_A
lr_A = 2e-4
lr_ratio = 8     # B learns 8x faster than A

optimizer = torch.optim.AdamW([
    {"params": lora_A_params, "lr": lr_A},
    {"params": lora_B_params, "lr": lr_A * lr_ratio},
    {"params": other_params,  "lr": lr_A},
], weight_decay=0.01)

3. Prefix Tuning & P-Tuning

3.1 Prefix Tuning

Prefix Tuning prepends a set of learnable "virtual tokens" to the key and value matrices at every attention layer. These prefix vectors are optimized during training while the rest of the model remains frozen. Because the prefixes operate entirely within the attention mechanism, they can steer model behavior without modifying any weight matrices.

The prefix length (number of virtual tokens) controls the method's capacity. Typical values range from 10 to 100. Longer prefixes allow more expressive adaptation but increase the effective sequence length and thus the computational cost of each forward pass.

from peft import PrefixTuningConfig, get_peft_model, TaskType

# Prefix Tuning configuration
prefix_config = PrefixTuningConfig(
    task_type=TaskType.CAUSAL_LM,
    num_virtual_tokens=30,        # Number of prefix tokens
    prefix_projection=True,       # Use MLP to project prefix (more stable)
    encoder_hidden_size=1024,     # Hidden size of projection MLP
)

model = get_peft_model(base_model, prefix_config)
model.print_trainable_parameters()
# Typically 0.1-0.5% of total parameters

3.2 Prompt Tuning

Prompt Tuning (from Google, 2021) is even simpler than Prefix Tuning: it prepends learnable embeddings to the input only at the embedding layer, not at every attention layer. This makes it the most parameter-efficient method (often under 0.01% of parameters) but limits its expressiveness. Prompt Tuning works best for large models (100B+), where a single model can serve many tasks by simply swapping the learned prompt prefix.

from peft import PromptTuningConfig, get_peft_model, TaskType
from peft import PromptTuningInit

# Prompt Tuning: learns soft tokens at input layer only
prompt_config = PromptTuningConfig(
    task_type=TaskType.CAUSAL_LM,
    num_virtual_tokens=20,
    prompt_tuning_init=PromptTuningInit.TEXT,  # Initialize from text
    prompt_tuning_init_text="Classify the following text as positive or negative:",
    tokenizer_name_or_path="meta-llama/Meta-Llama-3-8B",
)

model = get_peft_model(base_model, prompt_config)
model.print_trainable_parameters()
# trainable params: ~80K (extremely small)

4. Adapter Layers

Adapter methods (Houlsby et al., 2019) insert small bottleneck modules between existing transformer layers. Each adapter consists of a down-projection, a nonlinear activation, and an up-projection. The adapter bottleneck dimension controls the parameter count, similar to how rank works in LoRA.

Unlike LoRA (which modifies existing weight matrices), adapters add new layers to the network. This means they introduce a small amount of inference latency, because the adapter computation happens sequentially. However, adapters are very flexible and can be inserted at various points in the architecture.

from peft import AdaptionPromptConfig, get_peft_model

# Note: For bottleneck adapters, use the adapters library
# from adapters import AutoAdapterModel

# Example with LLaMA-Adapter style (via PEFT)
adapter_config = AdaptionPromptConfig(
    adapter_len=10,            # Length of adapter prompt
    adapter_layers=30,         # Number of layers to add adapters
    task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, adapter_config)
model.print_trainable_parameters()

5. IA3: Infused Adapter by Inhibiting and Amplifying Inner Activations

IA3 (few-shot parameter-efficient fine-tuning Is All You Need, 2022) takes parameter efficiency to the extreme. Instead of learning new matrices or inserting new layers, IA3 learns only three rescaling vectors that modulate the keys, values, and intermediate activations in the feedforward layers. The total number of trainable parameters is typically 10x smaller than LoRA.

The tradeoff is that IA3's limited capacity makes it best suited for simple adaptation tasks (format changes, style transfer) rather than complex domain adaptation. It excels in few-shot settings where overfitting is a concern.

from peft import IA3Config, get_peft_model, TaskType

ia3_config = IA3Config(
    task_type=TaskType.CAUSAL_LM,
    target_modules=["k_proj", "v_proj", "down_proj"],
    feedforward_modules=["down_proj"],
)

model = get_peft_model(base_model, ia3_config)
model.print_trainable_parameters()
# trainable params: ~500K for a 7B model (0.007%)

6. Comprehensive PEFT Method Comparison

7. Multi-Adapter Serving

One of LoRA's most powerful production features is the ability to serve many adapters from a single base model. This enables multi-tenant deployments where each customer or task gets its own fine-tuned behavior without duplicating the base model weights. Two main systems support this at scale: LoRAX (formerly LoRAX by Predibase) and S-LoRA.

7.1 Architecture Overview

7.2 LoRAX and S-LoRA

LoRAX (Predibase) is a production-grade serving system that can host hundreds of fine-tuned LoRA adapters on a single GPU. It keeps the base model in GPU memory and dynamically loads adapter weights per request. Key features include adapter weight caching, batched inference across different adapters, and automatic adapter management.

S-LoRA (from UC Berkeley) takes a more research-oriented approach, using unified paging to manage adapter memory and custom CUDA kernels for batched LoRA computation. S-LoRA can serve thousands of adapters simultaneously, with adapters stored in a tiered memory system (GPU, CPU, disk) and paged in on demand.

8. Choosing the Right PEFT Method

With so many PEFT options available, the decision can feel overwhelming. Here is a practical decision framework based on your constraints and requirements.