242 Parameter-Efficient Fine-Tuning: LoRA and QLoRA

242.1 Introduction

Full fine-tuning of a modern language model means computing a gradient for, and storing an optimizer state for, every one of its parameters. For a 7 billion parameter model in mixed precision, the weights alone occupy about 14 GB, and an Adam optimizer adds two more moment buffers per parameter, so the training-time memory footprint balloons to roughly 4 times the model size before activations are even counted. This puts full fine-tuning of frontier-scale models out of reach for anyone without a cluster, and it produces a fresh multi-gigabyte checkpoint for every task, which is unwieldy to store and serve.

Parameter-efficient fine-tuning (PEFT) attacks both problems at once. Instead of updating all the weights, it freezes the pretrained model and trains a small number of new parameters, often well under one percent of the total. The flagship method is LoRA (Low-Rank Adaptation), and its quantized extension QLoRA brought 65 billion parameter fine-tuning onto a single consumer GPU. This chapter develops the mathematics that makes a low-rank update both expressive enough to adapt a model and cheap enough to train, then shows production code: a from-scratch LoRA layer and the mature open-source peft library running on CPU, followed by the GPU-scale QLoRA recipe a reader can run on real hardware.

The whole field rests on mature, free, open-source tooling: Hugging Face peft, transformers, accelerate, and trl, together with bitsandbytes for quantization. None of it is proprietary, and the techniques transfer across model families.

242.2 The Core Idea: Updates Live on a Low-Rank Manifold

Fine-tuning takes a pretrained weight matrix $W_0 \in \mathbb{R}^{d \times k}$ and learns an update $\Delta W$, producing $W = W_0 + \Delta W$. Full fine-tuning lets $\Delta W$ range over all $d \times k$ matrices. The central empirical observation behind LoRA is that the update a downstream task actually needs has low intrinsic rank: the effective $\Delta W$ that adapts a large pretrained model to a new task can be approximated well by a matrix of rank $r \ll \min(d, k)$.

That observation is not arbitrary. Aghajanyan et al. (2021) showed that pretrained language models have a low intrinsic dimension: they can be fine-tuned to near full performance by optimizing in a randomly chosen low-dimensional subspace. LoRA turns this finding into a parameterization. Rather than learn a full $\Delta W$, write it as a product of two thin matrices,

\[ \Delta W = B A, \qquad B \in \mathbb{R}^{d \times r}, \quad A \in \mathbb{R}^{r \times k}, \]

so that the adapted forward pass is

\[ h = W_0 x + \Delta W x = W_0 x + B A x . \]

The frozen $W_0$ is never updated; only $A$ and $B$ are trained. The product $BA$ has rank at most $r$, which is exactly the low-rank constraint, and the number of trainable parameters drops from $d k$ to $r(d + k)$. For a $4096 \times 4096$ attention projection with $r = 8$, that is $65{,}536$ trainable values instead of $16.8$ million, a reduction of roughly 256 times for that matrix.

242.2.1 Scaling and Initialization

LoRA scales the low-rank term by a constant $\alpha / r$,

\[ h = W_0 x + \frac{\alpha}{r}\, B A x , \]

where $\alpha$ is a fixed hyperparameter. The reason is to decouple the learning-rate sensitivity from the choice of rank. If you double $r$, the typical magnitude of $BAx$ would otherwise grow, forcing you to retune the learning rate; dividing by $r$ keeps the update’s scale roughly constant across ranks, so $\alpha$ acts as a tunable gain and $r$ can be swept independently. A common convention is to set $\alpha = 2r$ and treat $\alpha/r$ as a constant of order one.

Initialization is chosen so that the adapter is the identity at the start of training. $A$ is initialized from a small random Gaussian and $B$ is initialized to zero, hence $\Delta W = BA = 0$ at step zero. The model therefore begins exactly at the pretrained function and departs from it smoothly as $B$ moves away from zero. Initializing both matrices randomly would inject noise into a carefully pretrained model on the very first forward pass, which hurts optimization.

242.2.2 Why It Saves So Much Memory

The dominant training-time cost is not the parameters themselves but the optimizer state and the gradients. Adam keeps two extra buffers (first and second moments) per trainable parameter. Because $W_0$ is frozen, it has no gradient and no optimizer state; only $A$ and $B$ do. If the adapters are one percent of the parameters, the optimizer state shrinks by roughly 100 times. The frozen base weights still occupy memory, but they can be held in lower precision (the QLoRA insight below), and they are read-only, so no gradient buffers are allocated for them.

242.2.3 Zero Inference Overhead by Merging

A property that sets LoRA apart from earlier adapter methods is that it adds no inference latency once training is done. Because the update is linear and additive, the learned adapter can be folded back into the base weight,

\[ W_{\text{merged}} = W_0 + \frac{\alpha}{r} B A , \]

producing a single matrix of the original shape. The deployed model is then byte-for-byte the same architecture as the base model, with no extra matrix multiplies at run time. This is why LoRA is the dominant serving-friendly PEFT method: bottleneck-style adapters that insert new layers add sequential compute at inference, whereas a merged LoRA does not. The flip side is that you can also keep adapters unmerged and hot-swap many task-specific $A, B$ pairs over one shared frozen base, which is the basis of multi-tenant LoRA serving.

242.3 QLoRA: Fine-Tuning a Quantized Base

LoRA shrinks the trainable parameters, but the frozen base model still has to sit in memory during training. For a 65 billion parameter model in 16-bit precision that is about 130 GB, still far beyond a single GPU. QLoRA (Dettmers et al., 2023) closes that gap with three ideas that let the frozen base be stored in 4-bit precision while LoRA adapters train in full precision on top.

4-bit NormalFloat (NF4). Standard integer quantization spaces its levels uniformly, which wastes resolution because neural network weights are approximately zero-mean Gaussian. NF4 is an information-theoretically optimal quantization for normally distributed data: its 16 levels are placed at the quantiles of a standard normal, so each level is equally likely to be used. The frozen weights are stored as NF4 codes and dequantized on the fly to a compute dtype (bfloat16) only for each matrix multiply.

Double quantization. Block-wise quantization stores one scaling constant per block of weights. Those constants themselves take space (about 0.5 bits per parameter for a block size of 64). Double quantization quantizes the quantization constants, saving roughly another 0.37 bits per parameter, which is meaningful at the 65 billion parameter scale.

Paged optimizers. Gradient checkpointing produces memory spikes that can overflow the GPU. QLoRA uses NVIDIA unified memory to page optimizer states to CPU RAM and back, smoothing those spikes instead of crashing.

The key correctness point is that gradients still flow. The forward pass dequantizes the 4-bit base weight to bfloat16, adds the bfloat16 LoRA term, and computes the loss; backpropagation updates only $A$ and $B$ (in bfloat16), never the quantized base. QLoRA showed this matches 16-bit full fine-tuning quality on instruction-following benchmarks while fitting a 65 billion parameter fine-tune on a single 48 GB GPU.

flowchart LR
    X["Input x"] --> Q["Frozen base W0 stored in 4-bit NF4"]
    X --> A["LoRA A trainable bf16"]
    Q --> DQ["Dequantize to bf16"]
    A --> B["LoRA B trainable bf16"]
    DQ --> S["Add"]
    B --> SC["Scale by alpha over r"]
    SC --> S
    S --> H["Output h"]

242.4 Production Code

The following two cells run on CPU and execute as part of the book build. They are deliberately tiny and deterministic so they finish in well under a minute.

242.4.1 A LoRA Linear Layer from Scratch

This cell implements the $W_0 + (\alpha/r) BA$ parameterization directly in PyTorch, freezes the base weight, and counts trainable parameters to make the reduction concrete.

Code

import torch
import torch.nn as nn

torch.manual_seed(0)


class LoRALinear(nn.Module):
    """A frozen Linear with an additive low-rank adapter B A scaled by alpha/r."""

    def __init__(self, in_features, out_features, r=4, alpha=8):
        super().__init__()
        # Frozen pretrained weight (stand-in for a pretrained checkpoint).
        self.base = nn.Linear(in_features, out_features, bias=True)
        self.base.weight.requires_grad_(False)
        self.base.bias.requires_grad_(False)

        self.r = r
        self.scaling = alpha / r
        # A is small-random, B is zero, so B A = 0 at initialization.
        self.A = nn.Parameter(torch.randn(r, in_features) * 0.01)
        self.B = nn.Parameter(torch.zeros(out_features, r))

    def forward(self, x):
        base_out = self.base(x)
        lora_out = (x @ self.A.t() @ self.B.t()) * self.scaling
        return base_out + lora_out


layer = LoRALinear(in_features=512, out_features=512, r=8, alpha=16)

trainable = sum(p.numel() for p in layer.parameters() if p.requires_grad)
frozen = sum(p.numel() for p in layer.parameters() if not p.requires_grad)
total = trainable + frozen

print(f"frozen base params : {frozen:,}")
print(f"trainable LoRA params: {trainable:,}")
print(f"trainable fraction   : {100 * trainable / total:.2f}%")

# At initialization the adapter is the identity (B = 0), so the LoRA layer
# matches the frozen base exactly.
x = torch.randn(3, 512)
with torch.no_grad():
    same = torch.allclose(layer(x), layer.base(x))
print(f"adapter is identity at init: {same}")

frozen base params : 262,656
trainable LoRA params: 8,192
trainable fraction   : 3.02%
adapter is identity at init: True

The frozen $512 \times 512$ weight has $262{,}144$ parameters; the rank-8 adapter adds only a few thousand, and the layer reproduces the base model exactly until training begins.

242.4.2 Training Only the Adapter

Here we fit the LoRA layer to a synthetic target on CPU, confirming that the frozen base never moves while the loss falls. Only $A$ and $B$ receive gradients.

Code

import time

torch.manual_seed(1)

# Synthetic task: the base model already computes W0 x; the target adds a
# genuinely low-rank shift on top of it, exactly the regime LoRA is built for.
model_for_data = LoRALinear(512, 512, r=8, alpha=16)
shift_U = torch.randn(512, 8)
shift_V = torch.randn(8, 512)
delta = (shift_U @ shift_V) / 8.0  # a rank-8 update to learn
X = torch.randn(256, 512)
with torch.no_grad():
    Y = model_for_data.base(X) + X @ delta.t() + 0.01 * torch.randn(256, 512)

# Train a fresh adapter over the same frozen base, so only the rank-8 shift
# remains to be learned.
model = model_for_data
opt = torch.optim.Adam([p for p in model.parameters() if p.requires_grad], lr=1e-2)
loss_fn = nn.MSELoss()

base_weight_before = model.base.weight.detach().clone()

start = time.time()
loss0 = loss_fn(model(X), Y).item()
for step in range(200):
    opt.zero_grad()
    loss = loss_fn(model(X), Y)
    loss.backward()
    opt.step()
elapsed = time.time() - start

base_unchanged = torch.equal(base_weight_before, model.base.weight.detach())

print(f"initial loss : {loss0:.4f}")
print(f"final loss   : {loss.item():.6f}")
print(f"frozen base weight unchanged: {base_unchanged}")
print(f"training time (200 steps, CPU): {elapsed:.2f}s")

initial loss : 64.0971
final loss   : 0.000096
frozen base weight unchanged: True
training time (200 steps, CPU): 2.07s

The target here is a genuinely rank-8 shift on top of the frozen base, so a rank-8 adapter can represent it exactly; the loss falls almost to zero while the base weight stays bit-for-bit identical before and after training. That is the defining behavior of LoRA: the pretrained knowledge is preserved and only the small adapter adapts. A real task’s update is not exactly low-rank, but the same mechanism captures the part that is, which is empirically most of what matters.

242.4.3 The peft Library on a Tiny Model

In production you do not hand-roll LoRA; you wrap an existing model with the open-source peft library, which injects adapters into named submodules and handles saving, loading, and merging. This cell applies peft to a tiny multilayer network on CPU and reports the trainable-parameter reduction it computes.

Code

from peft import LoraConfig, get_peft_model

torch.manual_seed(0)


class TinyMLP(nn.Module):
    def __init__(self, d_in=64, d_hidden=256, d_out=8):
        super().__init__()
        self.fc1 = nn.Linear(d_in, d_hidden)
        self.act = nn.ReLU()
        self.fc2 = nn.Linear(d_hidden, d_out)

    def forward(self, x):
        return self.fc2(self.act(self.fc1(x)))


base = TinyMLP()
dense_trainable = sum(p.numel() for p in base.parameters() if p.requires_grad)

config = LoraConfig(
    r=4,
    lora_alpha=8,
    target_modules=["fc1", "fc2"],  # inject LoRA into these named Linear layers
    lora_dropout=0.0,
    bias="none",
)
lora_model = get_peft_model(base, config)

lora_trainable = sum(p.numel() for p in lora_model.parameters() if p.requires_grad)

lora_model.print_trainable_parameters()
print(f"dense full fine-tuning would train: {dense_trainable:,} params")
print(f"LoRA trains instead              : {lora_trainable:,} params")
print(f"reduction factor                 : {dense_trainable / lora_trainable:.1f}x")

C:\Users\miken\github\ai_in_action\.venv\lib\site-packages\tqdm\auto.py:21: TqdmWarning:

IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html

trainable params: 2,336 || all params: 21,032 || trainable%: 11.1069
dense full fine-tuning would train: 18,696 params
LoRA trains instead              : 2,336 params
reduction factor                 : 8.0x

peft reports the same accounting our hand-rolled layer did: a small fraction of the parameters are trainable. The same three lines (LoraConfig, get_peft_model, train) apply unchanged to a 7 billion parameter transformers model; only target_modules and the trainer change.

242.4.4 QLoRA at Scale (requires GPU and a large model)

The next block is the real QLoRA recipe for a 7 billion parameter model. It is shown, not executed, because 4-bit bitsandbytes kernels require a CUDA GPU; running it on the CPU build is not feasible. A reader with a single 24 GB GPU can run it as written.

# Requires a CUDA GPU. Loads the base model in 4-bit NF4 and trains LoRA on top.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

model_id = "mistralai/Mistral-7B-v0.1"

# 4-bit NormalFloat with double quantization; compute in bfloat16.
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
)

# Casts layernorms to fp32 and enables gradient checkpointing for stability.
model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    # Adapt the attention and MLP projections, the standard QLoRA target set.
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # typically well under 1 percent trainable

Training then proceeds with the standard trl supervised fine-tuning trainer, which works identically whether the base is full precision or 4-bit:

# Requires a CUDA GPU. Standard SFT loop over the quantized base + LoRA adapters.
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

dataset = load_dataset("imdb", split="train[:1000]")

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    args=SFTConfig(
        output_dir="qlora-out",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        num_train_epochs=1,
        bf16=True,
        logging_steps=10,
    ),
)
trainer.train()

# Save just the adapter (a few megabytes), not the whole base model.
model.save_pretrained("qlora-adapter")

The saved artifact is only the adapter, often a few megabytes, so a single 4-bit base can be shared across many tasks, each with its own tiny adapter. For deployment you either keep adapters separate and hot-swap them, or call model.merge_and_unload() to fold a chosen adapter into the (dequantized) base for zero-overhead inference.

242.4.5 Running it on a GPU (executed)

The 7 billion parameter recipe above is shown rather than executed because it needs a large download and more VRAM than a typical workstation has free. To make QLoRA concrete on real hardware, the next cell runs the identical machinery on a small pretrained model that fits comfortably on a single 8 GB consumer GPU. It loads Qwen/Qwen2.5-0.5B-Instruct (falling back to gpt2 if that download is unavailable) in 4-bit NF4 with double quantization, attaches LoRA adapters with peft, prints the trainable-vs-total accounting, and runs one full forward, backward, and optimizer step on a tiny synthetic batch. It measures real VRAM and confirms the loss is finite and the adapter parameters actually move while the quantized base stays put. Everything is guarded by torch.cuda.is_available(), with a CPU fallback message, so the book still builds on a machine without a GPU.

Code

import torch

if torch.cuda.is_available():
    import time
    from transformers import (
        AutoModelForCausalLM,
        AutoTokenizer,
        BitsAndBytesConfig,
    )
    from peft import (
        LoraConfig,
        get_peft_model,
        prepare_model_for_kbit_training,
    )

    torch.manual_seed(0)
    torch.cuda.manual_seed_all(0)
    torch.cuda.reset_peak_memory_stats()

    device = torch.device("cuda")
    gpu_name = torch.cuda.get_device_name(0)

    # 4-bit NormalFloat with double quantization; compute in bfloat16. This is
    # the exact QLoRA configuration used at 7B scale, just on a small base.
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True,
        bnb_4bit_compute_dtype=torch.bfloat16,
    )

    # Prefer a small instruction-tuned model; fall back to gpt2 if unavailable.
    candidate_ids = ["Qwen/Qwen2.5-0.5B-Instruct", "gpt2"]
    model = None
    model_id = None
    for candidate in candidate_ids:
        try:
            tokenizer = AutoTokenizer.from_pretrained(candidate)
            model = AutoModelForCausalLM.from_pretrained(
                candidate,
                quantization_config=bnb_config,
                device_map={"": 0},
            )
            model_id = candidate
            break
        except Exception as exc:  # download or load failure: try the next one
            print(f"could not load {candidate}: {type(exc).__name__}")

    if model is None:
        print("no base model could be loaded; skipping GPU QLoRA demo")
    else:
        if tokenizer.pad_token is None:
            tokenizer.pad_token = tokenizer.eos_token

        load_mem_gb = torch.cuda.memory_allocated() / 1e9
        print(f"GPU                 : {gpu_name}")
        print(f"base model          : {model_id}")
        print(f"VRAM after 4-bit load: {load_mem_gb:.3f} GB")

        # Cast layernorms to fp32 and enable gradient checkpointing for stability,
        # then attach LoRA adapters over the standard projection set.
        model = prepare_model_for_kbit_training(model)
        lora_config = LoraConfig(
            r=8,
            lora_alpha=16,
            lora_dropout=0.0,
            bias="none",
            task_type="CAUSAL_LM",
        )
        model = get_peft_model(model, lora_config)

        trainable = sum(
            p.numel() for p in model.parameters() if p.requires_grad
        )
        total = sum(p.numel() for p in model.parameters())
        print(f"trainable params    : {trainable:,}")
        print(f"total params        : {total:,}")
        print(f"trainable fraction  : {100 * trainable / total:.4f}%")

        # Tiny synthetic batch: a couple of short prompts, labels = inputs so the
        # causal LM loss is well defined. Short sequences keep memory minimal.
        prompts = [
            "Low-rank adaptation freezes the base model and trains",
            "Quantized fine-tuning stores the frozen weights in",
        ]
        batch = tokenizer(
            prompts,
            return_tensors="pt",
            padding=True,
            truncation=True,
            max_length=32,
        ).to(device)
        labels = batch["input_ids"].clone()

        # Snapshot one adapter tensor to confirm it actually updates.
        lora_param = next(
            p for n, p in model.named_parameters()
            if p.requires_grad and "lora" in n.lower()
        )
        before = lora_param.detach().clone()

        opt = torch.optim.AdamW(
            [p for p in model.parameters() if p.requires_grad], lr=2e-4
        )

        model.train()
        torch.cuda.synchronize()
        start = time.time()

        opt.zero_grad()
        out = model(**batch, labels=labels)
        loss = out.loss
        loss.backward()
        opt.step()

        torch.cuda.synchronize()
        step_ms = (time.time() - start) * 1e3

        moved = not torch.equal(before, lora_param.detach())
        peak_mem_gb = torch.cuda.max_memory_allocated() / 1e9

        print(f"step loss           : {loss.item():.4f}")
        print(f"loss is finite      : {torch.isfinite(loss).item()}")
        print(f"adapter params moved: {moved}")
        print(f"forward+backward+step: {step_ms:.1f} ms")
        print(f"peak VRAM           : {peak_mem_gb:.3f} GB")

    # Release everything so later cells and other processes get the VRAM back.
    del model
    torch.cuda.empty_cache()
else:
    print("CUDA not available; skipping the GPU QLoRA demo.")
    print("On a CUDA machine this cell runs real 4-bit QLoRA on a small model.")

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Loading weights:   0%|          | 0/290 [00:00<?, ?it/s]Loading weights:   0%|          | 1/290 [00:00<00:47,  6.05it/s]Loading weights:   1%|          | 3/290 [00:00<00:23, 12.16it/s]Loading weights:  14%|█▍        | 41/290 [00:00<00:01, 155.66it/s]Loading weights:  30%|███       | 88/290 [00:00<00:00, 257.76it/s]Loading weights:  43%|████▎     | 126/290 [00:00<00:00, 294.19it/s]Loading weights:  58%|█████▊    | 167/290 [00:00<00:00, 328.54it/s]Loading weights:  72%|███████▏  | 209/290 [00:00<00:00, 349.41it/s]Loading weights:  87%|████████▋ | 251/290 [00:00<00:00, 368.82it/s]Loading weights: 100%|█████████▉| 289/290 [00:01<00:00, 368.55it/s]Loading weights: 100%|██████████| 290/290 [00:01<00:00, 288.52it/s]

GPU                 : NVIDIA GeForce RTX 3070 Ti Laptop GPU
base model          : Qwen/Qwen2.5-0.5B-Instruct
VRAM after 4-bit load: 0.457 GB
trainable params    : 540,672
total params        : 315,660,160
trainable fraction  : 0.1713%

[transformers] `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
C:\Users\miken\github\ai_in_action\.venv\lib\site-packages\torch\_dynamo\eval_frame.py:745: UserWarning:

torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.5 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.

step loss           : 6.2574
loss is finite      : True
adapter params moved: True
forward+backward+step: 1249.6 ms
peak VRAM           : 1.003 GB

This is genuine QLoRA on the GPU: the frozen base is held in 4-bit NF4, only the bfloat16 LoRA adapters carry gradients, and the printed peak VRAM (well under 2 GB for the 0.5B base) shows why the method fits where full fine-tuning never could. The same code scales straight up to the 7 billion parameter recipe above by swapping the model id and adding the trl trainer; the only reason that larger run is not executed here is the download size and VRAM headroom, not any difference in the technique.

The blocks that follow stay show-only for honest reasons specific to this environment. Multi-GPU FSDP sharding needs more than one device, large-model RLHF needs far more VRAM than a single 8 GB card, and FlashAttention’s prebuilt kernels are not reliably available on Windows; each is noted as such rather than faked.

# Show-only: multi-GPU FSDP QLoRA. Requires 2+ GPUs and accelerate launch,
# so it cannot run on a single-GPU workstation.
from accelerate import Accelerator

accelerator = Accelerator()  # picks up an FSDP config from `accelerate config`
model, opt, loader = accelerator.prepare(model, opt, loader)
# accelerate launch --num_processes 4 --use_fsdp train_qlora.py

# Show-only: FlashAttention-2 attention kernels. The prebuilt wheels target
# Linux+CUDA; on Windows the install is unreliable, so this stays unexecuted.
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    attn_implementation="flash_attention_2",  # Linux/CUDA only
    device_map={"": 0},
)

242.5 Pitfalls and When to Use

Choosing the rank. Rank $r$ is the main capability knob. Ranks of 8 to 16 are strong defaults and often match full fine-tuning on instruction tasks. Pushing $r$ higher rarely helps and wastes parameters once the update’s intrinsic rank is covered; pushing it too low can underfit a task that genuinely changes the model’s behavior. Sweep $r$ on a validation set rather than guessing.

Which modules to target. The original LoRA paper adapted only the attention query and value projections. Modern practice, especially QLoRA, targets all linear projections including the MLP, which usually helps and costs little. Targeting too few modules (for example only q_proj) is a common cause of underfitting; if a LoRA run plateaus below expectations, widening target_modules is the first thing to try.

The alpha-over-r coupling. Because the effective gain is $\alpha/r$, changing $r$ without changing $\alpha$ silently changes the update magnitude. Keep $\alpha/r$ fixed (for example $\alpha = 2r$) when sweeping rank, or you will conflate the effect of rank with the effect of learning rate.

Quantization is not free quality. QLoRA’s 4-bit base introduces quantization error. On most instruction-tuning tasks this is negligible, but for tasks demanding high numerical precision, or when the base is small enough to fit in 16-bit anyway, plain LoRA (or full fine-tuning) can be the better choice. Quantize because you must fit memory, not reflexively.

Merging and serving trade-offs. Merging gives zero inference overhead but bakes in one adapter; if you need many tasks on one base, keep adapters unmerged and pay a small per-call cost to swap them. Merging a LoRA trained on a 4-bit base requires dequantizing first, which can reintroduce a small accuracy shift, so validate the merged model rather than assuming it is identical to the unmerged one.

When PEFT is the wrong tool. LoRA adapts behavior; it does not efficiently inject large amounts of new factual knowledge or change a model’s tokenizer or architecture. If a task requires the model to learn a substantial new domain vocabulary or capability that the base genuinely lacks, continued pretraining or full fine-tuning may be necessary. PEFT shines for instruction tuning, style and persona adaptation, task specialization, and multi-tenant serving, which together cover the large majority of practical fine-tuning needs.

242.6 Conclusion

LoRA rests on a single sharp observation: the weight update that adapts a large pretrained model to a downstream task lives on a low-rank manifold, so it can be parameterized as a thin product $BA$ with a few thousand trainable values per matrix instead of millions. That cuts optimizer memory by orders of magnitude, produces tiny shareable checkpoints, and, because the update is linear, merges back into the base for zero inference overhead. QLoRA extends the idea downward in precision, storing the frozen base in 4-bit NormalFloat while training full-precision adapters on top, which brought billion-parameter fine-tuning onto a single GPU. The executed CPU code showed the parameter accounting and the frozen-base invariant directly; the GPU recipe showed the exact, runnable invocation against the mature open-source stack. For the large majority of fine-tuning work today, a rank-16 LoRA over a quantized base is the default, and understanding why it works is now prerequisite to doing it well.

242.7 References

Hu, E. et al. “LoRA: Low-Rank Adaptation of Large Language Models.” ICLR 2022. https://arxiv.org/abs/2106.09685
Dettmers, T. et al. “QLoRA: Efficient Finetuning of Quantized LLMs.” NeurIPS 2023. https://arxiv.org/abs/2305.14314
Aghajanyan, A., Zettlemoyer, L., Gupta, S. “Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning.” ACL 2021. https://arxiv.org/abs/2012.13255
Houlsby, N. et al. “Parameter-Efficient Transfer Learning for NLP” (bottleneck adapters). ICML 2019. https://arxiv.org/abs/1902.00751
Mangrulkar, S. et al. “PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods.” Hugging Face, 2022. https://github.com/huggingface/peft
Dettmers, T. et al. “8-bit Optimizers via Block-wise Quantization” (bitsandbytes). ICLR 2022. https://arxiv.org/abs/2110.02861

# Parameter-Efficient Fine-Tuning: LoRA and QLoRA {#sec-lora-qlora} ## Introduction Full fine-tuning of a modern language model means computing a gradient for, and storing an optimizer state for, every one of its parameters. For a 7 billion parameter model in mixed precision, the weights alone occupy about 14 GB, and an Adam optimizer adds two more moment buffers per parameter, so the training-time memory footprint balloons to roughly 4 times the model size before activations are even counted. This puts full fine-tuning of frontier-scale models out of reach for anyone without a cluster, and it produces a fresh multi-gigabyte checkpoint for every task, which is unwieldy to store and serve. Parameter-efficient fine-tuning (PEFT) attacks both problems at once. Instead of updating all the weights, it freezes the pretrained model and trains a small number of new parameters, often well under one percent of the total. The flagship method is **LoRA** (Low-Rank Adaptation), and its quantized extension **QLoRA** brought 65 billion parameter fine-tuning onto a single consumer GPU. This chapter develops the mathematics that makes a low-rank update both expressive enough to adapt a model and cheap enough to train, then shows production code: a from-scratch LoRA layer and the mature open-source `peft` library running on CPU, followed by the GPU-scale QLoRA recipe a reader can run on real hardware. The whole field rests on mature, free, open-source tooling: Hugging Face `peft`, `transformers`, `accelerate`, and `trl`, together with `bitsandbytes` for quantization. None of it is proprietary, and the techniques transfer across model families. ## The Core Idea: Updates Live on a Low-Rank Manifold Fine-tuning takes a pretrained weight matrix $W_0 \in \mathbb{R}^{d \times k}$ and learns an update $\Delta W$, producing $W = W_0 + \Delta W$. Full fine-tuning lets $\Delta W$ range over all $d \times k$ matrices. The central empirical observation behind LoRA is that the update a downstream task actually needs has **low intrinsic rank**: the effective $\Delta W$ that adapts a large pretrained model to a new task can be approximated well by a matrix of rank $r \ll \min(d, k)$. That observation is not arbitrary. Aghajanyan et al. (2021) showed that pretrained language models have a low *intrinsic dimension*: they can be fine-tuned to near full performance by optimizing in a randomly chosen low-dimensional subspace. LoRA turns this finding into a parameterization. Rather than learn a full $\Delta W$, write it as a product of two thin matrices, $$ \Delta W = B A, \qquad B \in \mathbb{R}^{d \times r}, \quad A \in \mathbb{R}^{r \times k}, $$ so that the adapted forward pass is $$ h = W_0 x + \Delta W x = W_0 x + B A x . $$ The frozen $W_0$ is never updated; only $A$ and $B$ are trained. The product $BA$ has rank at most $r$, which is exactly the low-rank constraint, and the number of trainable parameters drops from $d k$ to $r(d + k)$. For a $4096 \times 4096$ attention projection with $r = 8$, that is $65{,}536$ trainable values instead of $16.8$ million, a reduction of roughly 256 times for that matrix. ### Scaling and Initialization LoRA scales the low-rank term by a constant $\alpha / r$, $$ h = W_0 x + \frac{\alpha}{r}\, B A x , $$ where $\alpha$ is a fixed hyperparameter. The reason is to decouple the learning-rate sensitivity from the choice of rank. If you double $r$, the typical magnitude of $BAx$ would otherwise grow, forcing you to retune the learning rate; dividing by $r$ keeps the update's scale roughly constant across ranks, so $\alpha$ acts as a tunable gain and $r$ can be swept independently. A common convention is to set $\alpha = 2r$ and treat $\alpha/r$ as a constant of order one. Initialization is chosen so that the adapter is the identity at the start of training. $A$ is initialized from a small random Gaussian and $B$ is initialized to zero, hence $\Delta W = BA = 0$ at step zero. The model therefore begins exactly at the pretrained function and departs from it smoothly as $B$ moves away from zero. Initializing both matrices randomly would inject noise into a carefully pretrained model on the very first forward pass, which hurts optimization. ### Why It Saves So Much Memory The dominant training-time cost is not the parameters themselves but the optimizer state and the gradients. Adam keeps two extra buffers (first and second moments) per *trainable* parameter. Because $W_0$ is frozen, it has no gradient and no optimizer state; only $A$ and $B$ do. If the adapters are one percent of the parameters, the optimizer state shrinks by roughly 100 times. The frozen base weights still occupy memory, but they can be held in lower precision (the QLoRA insight below), and they are read-only, so no gradient buffers are allocated for them. ### Zero Inference Overhead by Merging A property that sets LoRA apart from earlier adapter methods is that it adds **no inference latency** once training is done. Because the update is linear and additive, the learned adapter can be folded back into the base weight, $$ W_{\text{merged}} = W_0 + \frac{\alpha}{r} B A , $$ producing a single matrix of the original shape. The deployed model is then byte-for-byte the same architecture as the base model, with no extra matrix multiplies at run time. This is why LoRA is the dominant serving-friendly PEFT method: bottleneck-style adapters that insert new layers add sequential compute at inference, whereas a merged LoRA does not. The flip side is that you can also keep adapters *unmerged* and hot-swap many task-specific $A, B$ pairs over one shared frozen base, which is the basis of multi-tenant LoRA serving. ## QLoRA: Fine-Tuning a Quantized Base LoRA shrinks the trainable parameters, but the frozen base model still has to sit in memory during training. For a 65 billion parameter model in 16-bit precision that is about 130 GB, still far beyond a single GPU. **QLoRA** (Dettmers et al., 2023) closes that gap with three ideas that let the frozen base be stored in 4-bit precision while LoRA adapters train in full precision on top. **4-bit NormalFloat (NF4).** Standard integer quantization spaces its levels uniformly, which wastes resolution because neural network weights are approximately zero-mean Gaussian. NF4 is an *information-theoretically optimal* quantization for normally distributed data: its 16 levels are placed at the quantiles of a standard normal, so each level is equally likely to be used. The frozen weights are stored as NF4 codes and dequantized on the fly to a compute dtype (bfloat16) only for each matrix multiply. **Double quantization.** Block-wise quantization stores one scaling constant per block of weights. Those constants themselves take space (about 0.5 bits per parameter for a block size of 64). Double quantization quantizes the quantization constants, saving roughly another 0.37 bits per parameter, which is meaningful at the 65 billion parameter scale. **Paged optimizers.** Gradient checkpointing produces memory spikes that can overflow the GPU. QLoRA uses NVIDIA unified memory to page optimizer states to CPU RAM and back, smoothing those spikes instead of crashing. The key correctness point is that gradients still flow. The forward pass dequantizes the 4-bit base weight to bfloat16, adds the bfloat16 LoRA term, and computes the loss; backpropagation updates only $A$ and $B$ (in bfloat16), never the quantized base. QLoRA showed this matches 16-bit full fine-tuning quality on instruction-following benchmarks while fitting a 65 billion parameter fine-tune on a single 48 GB GPU. ```{mermaid} flowchart LR X["Input x"] --> Q["Frozen base W0 stored in 4-bit NF4"] X --> A["LoRA A trainable bf16"] Q --> DQ["Dequantize to bf16"] A --> B["LoRA B trainable bf16"] DQ --> S["Add"] B --> SC["Scale by alpha over r"] SC --> S S --> H["Output h"] ``` ## Production Code The following two cells run on CPU and execute as part of the book build. They are deliberately tiny and deterministic so they finish in well under a minute. ### A LoRA Linear Layer from Scratch This cell implements the $W_0 + (\alpha/r) BA$ parameterization directly in PyTorch, freezes the base weight, and counts trainable parameters to make the reduction concrete. ```{python} import torch import torch.nn as nn torch.manual_seed(0) class LoRALinear(nn.Module): """A frozen Linear with an additive low-rank adapter B A scaled by alpha/r.""" def __init__(self, in_features, out_features, r=4, alpha=8): super().__init__() # Frozen pretrained weight (stand-in for a pretrained checkpoint). self.base = nn.Linear(in_features, out_features, bias=True) self.base.weight.requires_grad_(False) self.base.bias.requires_grad_(False) self.r = r self.scaling = alpha / r # A is small-random, B is zero, so B A = 0 at initialization. self.A = nn.Parameter(torch.randn(r, in_features) * 0.01) self.B = nn.Parameter(torch.zeros(out_features, r)) def forward(self, x): base_out = self.base(x) lora_out = (x @ self.A.t() @ self.B.t()) * self.scaling return base_out + lora_out layer = LoRALinear(in_features=512, out_features=512, r=8, alpha=16) trainable = sum(p.numel() for p in layer.parameters() if p.requires_grad) frozen = sum(p.numel() for p in layer.parameters() if not p.requires_grad) total = trainable + frozen print(f"frozen base params : {frozen:,}") print(f"trainable LoRA params: {trainable:,}") print(f"trainable fraction : {100 * trainable / total:.2f}%") # At initialization the adapter is the identity (B = 0), so the LoRA layer # matches the frozen base exactly. x = torch.randn(3, 512) with torch.no_grad(): same = torch.allclose(layer(x), layer.base(x)) print(f"adapter is identity at init: {same}") ``` The frozen $512 \times 512$ weight has $262{,}144$ parameters; the rank-8 adapter adds only a few thousand, and the layer reproduces the base model exactly until training begins. ### Training Only the Adapter Here we fit the LoRA layer to a synthetic target on CPU, confirming that the frozen base never moves while the loss falls. Only $A$ and $B$ receive gradients. ```{python} import time torch.manual_seed(1) # Synthetic task: the base model already computes W0 x; the target adds a # genuinely low-rank shift on top of it, exactly the regime LoRA is built for. model_for_data = LoRALinear(512, 512, r=8, alpha=16) shift_U = torch.randn(512, 8) shift_V = torch.randn(8, 512) delta = (shift_U @ shift_V) / 8.0 # a rank-8 update to learn X = torch.randn(256, 512) with torch.no_grad(): Y = model_for_data.base(X) + X @ delta.t() + 0.01 * torch.randn(256, 512) # Train a fresh adapter over the same frozen base, so only the rank-8 shift # remains to be learned. model = model_for_data opt = torch.optim.Adam([p for p in model.parameters() if p.requires_grad], lr=1e-2) loss_fn = nn.MSELoss() base_weight_before = model.base.weight.detach().clone() start = time.time() loss0 = loss_fn(model(X), Y).item() for step in range(200): opt.zero_grad() loss = loss_fn(model(X), Y) loss.backward() opt.step() elapsed = time.time() - start base_unchanged = torch.equal(base_weight_before, model.base.weight.detach()) print(f"initial loss : {loss0:.4f}") print(f"final loss : {loss.item():.6f}") print(f"frozen base weight unchanged: {base_unchanged}") print(f"training time (200 steps, CPU): {elapsed:.2f}s") ``` The target here is a genuinely rank-8 shift on top of the frozen base, so a rank-8 adapter can represent it exactly; the loss falls almost to zero while the base weight stays bit-for-bit identical before and after training. That is the defining behavior of LoRA: the pretrained knowledge is preserved and only the small adapter adapts. A real task's update is not exactly low-rank, but the same mechanism captures the part that is, which is empirically most of what matters. ### The peft Library on a Tiny Model In production you do not hand-roll LoRA; you wrap an existing model with the open-source `peft` library, which injects adapters into named submodules and handles saving, loading, and merging. This cell applies `peft` to a tiny multilayer network on CPU and reports the trainable-parameter reduction it computes. ```{python} from peft import LoraConfig, get_peft_model torch.manual_seed(0) class TinyMLP(nn.Module): def __init__(self, d_in=64, d_hidden=256, d_out=8): super().__init__() self.fc1 = nn.Linear(d_in, d_hidden) self.act = nn.ReLU() self.fc2 = nn.Linear(d_hidden, d_out) def forward(self, x): return self.fc2(self.act(self.fc1(x))) base = TinyMLP() dense_trainable = sum(p.numel() for p in base.parameters() if p.requires_grad) config = LoraConfig( r=4, lora_alpha=8, target_modules=["fc1", "fc2"], # inject LoRA into these named Linear layers lora_dropout=0.0, bias="none", ) lora_model = get_peft_model(base, config) lora_trainable = sum(p.numel() for p in lora_model.parameters() if p.requires_grad) lora_model.print_trainable_parameters() print(f"dense full fine-tuning would train: {dense_trainable:,} params") print(f"LoRA trains instead : {lora_trainable:,} params") print(f"reduction factor : {dense_trainable / lora_trainable:.1f}x") ``` `peft` reports the same accounting our hand-rolled layer did: a small fraction of the parameters are trainable. The same three lines (`LoraConfig`, `get_peft_model`, train) apply unchanged to a 7 billion parameter `transformers` model; only `target_modules` and the trainer change. ### QLoRA at Scale (requires GPU and a large model) The next block is the real QLoRA recipe for a 7 billion parameter model. It is shown, not executed, because 4-bit `bitsandbytes` kernels require a CUDA GPU; running it on the CPU build is not feasible. A reader with a single 24 GB GPU can run it as written. ```python # Requires a CUDA GPU. Loads the base model in 4-bit NF4 and trains LoRA on top. import torch from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training model_id = "mistralai/Mistral-7B-v0.1" # 4-bit NormalFloat with double quantization; compute in bfloat16. bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16, ) tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, quantization_config=bnb_config, device_map="auto", ) # Casts layernorms to fp32 and enables gradient checkpointing for stability. model = prepare_model_for_kbit_training(model) lora_config = LoraConfig( r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", # Adapt the attention and MLP projections, the standard QLoRA target set. target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], ) model = get_peft_model(model, lora_config) model.print_trainable_parameters() # typically well under 1 percent trainable ``` Training then proceeds with the standard `trl` supervised fine-tuning trainer, which works identically whether the base is full precision or 4-bit: ```python # Requires a CUDA GPU. Standard SFT loop over the quantized base + LoRA adapters. from trl import SFTTrainer, SFTConfig from datasets import load_dataset dataset = load_dataset("imdb", split="train[:1000]") trainer = SFTTrainer( model=model, train_dataset=dataset, args=SFTConfig( output_dir="qlora-out", per_device_train_batch_size=4, gradient_accumulation_steps=4, learning_rate=2e-4, num_train_epochs=1, bf16=True, logging_steps=10, ), ) trainer.train() # Save just the adapter (a few megabytes), not the whole base model. model.save_pretrained("qlora-adapter") ``` The saved artifact is only the adapter, often a few megabytes, so a single 4-bit base can be shared across many tasks, each with its own tiny adapter. For deployment you either keep adapters separate and hot-swap them, or call `model.merge_and_unload()` to fold a chosen adapter into the (dequantized) base for zero-overhead inference. ### Running it on a GPU (executed) The 7 billion parameter recipe above is shown rather than executed because it needs a large download and more VRAM than a typical workstation has free. To make QLoRA concrete on real hardware, the next cell runs the identical machinery on a small pretrained model that fits comfortably on a single 8 GB consumer GPU. It loads `Qwen/Qwen2.5-0.5B-Instruct` (falling back to `gpt2` if that download is unavailable) in 4-bit NF4 with double quantization, attaches LoRA adapters with `peft`, prints the trainable-vs-total accounting, and runs one full forward, backward, and optimizer step on a tiny synthetic batch. It measures real VRAM and confirms the loss is finite and the adapter parameters actually move while the quantized base stays put. Everything is guarded by `torch.cuda.is_available()`, with a CPU fallback message, so the book still builds on a machine without a GPU. ```{python} import torch if torch.cuda.is_available(): import time from transformers import ( AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, ) from peft import ( LoraConfig, get_peft_model, prepare_model_for_kbit_training, ) torch.manual_seed(0) torch.cuda.manual_seed_all(0) torch.cuda.reset_peak_memory_stats() device = torch.device("cuda") gpu_name = torch.cuda.get_device_name(0) # 4-bit NormalFloat with double quantization; compute in bfloat16. This is # the exact QLoRA configuration used at 7B scale, just on a small base. bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16, ) # Prefer a small instruction-tuned model; fall back to gpt2 if unavailable. candidate_ids = ["Qwen/Qwen2.5-0.5B-Instruct", "gpt2"] model = None model_id = None for candidate in candidate_ids: try: tokenizer = AutoTokenizer.from_pretrained(candidate) model = AutoModelForCausalLM.from_pretrained( candidate, quantization_config=bnb_config, device_map={"": 0}, ) model_id = candidate break except Exception as exc: # download or load failure: try the next one print(f"could not load {candidate}: {type(exc).__name__}") if model is None: print("no base model could be loaded; skipping GPU QLoRA demo") else: if tokenizer.pad_token is None: tokenizer.pad_token = tokenizer.eos_token load_mem_gb = torch.cuda.memory_allocated() / 1e9 print(f"GPU : {gpu_name}") print(f"base model : {model_id}") print(f"VRAM after 4-bit load: {load_mem_gb:.3f} GB") # Cast layernorms to fp32 and enable gradient checkpointing for stability, # then attach LoRA adapters over the standard projection set. model = prepare_model_for_kbit_training(model) lora_config = LoraConfig( r=8, lora_alpha=16, lora_dropout=0.0, bias="none", task_type="CAUSAL_LM", ) model = get_peft_model(model, lora_config) trainable = sum( p.numel() for p in model.parameters() if p.requires_grad ) total = sum(p.numel() for p in model.parameters()) print(f"trainable params : {trainable:,}") print(f"total params : {total:,}") print(f"trainable fraction : {100 * trainable / total:.4f}%") # Tiny synthetic batch: a couple of short prompts, labels = inputs so the # causal LM loss is well defined. Short sequences keep memory minimal. prompts = [ "Low-rank adaptation freezes the base model and trains", "Quantized fine-tuning stores the frozen weights in", ] batch = tokenizer( prompts, return_tensors="pt", padding=True, truncation=True, max_length=32, ).to(device) labels = batch["input_ids"].clone() # Snapshot one adapter tensor to confirm it actually updates. lora_param = next( p for n, p in model.named_parameters() if p.requires_grad and "lora" in n.lower() ) before = lora_param.detach().clone() opt = torch.optim.AdamW( [p for p in model.parameters() if p.requires_grad], lr=2e-4 ) model.train() torch.cuda.synchronize() start = time.time() opt.zero_grad() out = model(**batch, labels=labels) loss = out.loss loss.backward() opt.step() torch.cuda.synchronize() step_ms = (time.time() - start) * 1e3 moved = not torch.equal(before, lora_param.detach()) peak_mem_gb = torch.cuda.max_memory_allocated() / 1e9 print(f"step loss : {loss.item():.4f}") print(f"loss is finite : {torch.isfinite(loss).item()}") print(f"adapter params moved: {moved}") print(f"forward+backward+step: {step_ms:.1f} ms") print(f"peak VRAM : {peak_mem_gb:.3f} GB") # Release everything so later cells and other processes get the VRAM back. del model torch.cuda.empty_cache() else: print("CUDA not available; skipping the GPU QLoRA demo.") print("On a CUDA machine this cell runs real 4-bit QLoRA on a small model.") ``` This is genuine QLoRA on the GPU: the frozen base is held in 4-bit NF4, only the bfloat16 LoRA adapters carry gradients, and the printed peak VRAM (well under 2 GB for the 0.5B base) shows why the method fits where full fine-tuning never could. The same code scales straight up to the 7 billion parameter recipe above by swapping the model id and adding the `trl` trainer; the only reason that larger run is not executed here is the download size and VRAM headroom, not any difference in the technique. The blocks that follow stay show-only for honest reasons specific to this environment. Multi-GPU FSDP sharding needs more than one device, large-model RLHF needs far more VRAM than a single 8 GB card, and FlashAttention's prebuilt kernels are not reliably available on Windows; each is noted as such rather than faked. ```python # Show-only: multi-GPU FSDP QLoRA. Requires 2+ GPUs and accelerate launch, # so it cannot run on a single-GPU workstation. from accelerate import Accelerator accelerator = Accelerator() # picks up an FSDP config from `accelerate config` model, opt, loader = accelerator.prepare(model, opt, loader) # accelerate launch --num_processes 4 --use_fsdp train_qlora.py ``` ```python # Show-only: FlashAttention-2 attention kernels. The prebuilt wheels target # Linux+CUDA; on Windows the install is unreliable, so this stays unexecuted. model = AutoModelForCausalLM.from_pretrained( model_id, quantization_config=bnb_config, attn_implementation="flash_attention_2", # Linux/CUDA only device_map={"": 0}, ) ``` ## Pitfalls and When to Use **Choosing the rank.** Rank $r$ is the main capability knob. Ranks of 8 to 16 are strong defaults and often match full fine-tuning on instruction tasks. Pushing $r$ higher rarely helps and wastes parameters once the update's intrinsic rank is covered; pushing it too low can underfit a task that genuinely changes the model's behavior. Sweep $r$ on a validation set rather than guessing. **Which modules to target.** The original LoRA paper adapted only the attention query and value projections. Modern practice, especially QLoRA, targets all linear projections including the MLP, which usually helps and costs little. Targeting too few modules (for example only `q_proj`) is a common cause of underfitting; if a LoRA run plateaus below expectations, widening `target_modules` is the first thing to try. **The alpha-over-r coupling.** Because the effective gain is $\alpha/r$, changing $r$ without changing $\alpha$ silently changes the update magnitude. Keep $\alpha/r$ fixed (for example $\alpha = 2r$) when sweeping rank, or you will conflate the effect of rank with the effect of learning rate. **Quantization is not free quality.** QLoRA's 4-bit base introduces quantization error. On most instruction-tuning tasks this is negligible, but for tasks demanding high numerical precision, or when the base is small enough to fit in 16-bit anyway, plain LoRA (or full fine-tuning) can be the better choice. Quantize because you must fit memory, not reflexively. **Merging and serving trade-offs.** Merging gives zero inference overhead but bakes in one adapter; if you need many tasks on one base, keep adapters unmerged and pay a small per-call cost to swap them. Merging a LoRA trained on a 4-bit base requires dequantizing first, which can reintroduce a small accuracy shift, so validate the merged model rather than assuming it is identical to the unmerged one. **When PEFT is the wrong tool.** LoRA adapts behavior; it does not efficiently inject large amounts of new factual knowledge or change a model's tokenizer or architecture. If a task requires the model to learn a substantial new domain vocabulary or capability that the base genuinely lacks, continued pretraining or full fine-tuning may be necessary. PEFT shines for instruction tuning, style and persona adaptation, task specialization, and multi-tenant serving, which together cover the large majority of practical fine-tuning needs. ## Conclusion LoRA rests on a single sharp observation: the weight update that adapts a large pretrained model to a downstream task lives on a low-rank manifold, so it can be parameterized as a thin product $BA$ with a few thousand trainable values per matrix instead of millions. That cuts optimizer memory by orders of magnitude, produces tiny shareable checkpoints, and, because the update is linear, merges back into the base for zero inference overhead. QLoRA extends the idea downward in precision, storing the frozen base in 4-bit NormalFloat while training full-precision adapters on top, which brought billion-parameter fine-tuning onto a single GPU. The executed CPU code showed the parameter accounting and the frozen-base invariant directly; the GPU recipe showed the exact, runnable invocation against the mature open-source stack. For the large majority of fine-tuning work today, a rank-16 LoRA over a quantized base is the default, and understanding why it works is now prerequisite to doing it well. ## References 1. Hu, E. et al. "LoRA: Low-Rank Adaptation of Large Language Models." ICLR 2022. https://arxiv.org/abs/2106.09685 2. Dettmers, T. et al. "QLoRA: Efficient Finetuning of Quantized LLMs." NeurIPS 2023. https://arxiv.org/abs/2305.14314 3. Aghajanyan, A., Zettlemoyer, L., Gupta, S. "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning." ACL 2021. https://arxiv.org/abs/2012.13255 4. Houlsby, N. et al. "Parameter-Efficient Transfer Learning for NLP" (bottleneck adapters). ICML 2019. https://arxiv.org/abs/1902.00751 5. Mangrulkar, S. et al. "PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods." Hugging Face, 2022. https://github.com/huggingface/peft 6. Dettmers, T. et al. "8-bit Optimizers via Block-wise Quantization" (bitsandbytes). ICLR 2022. https://arxiv.org/abs/2110.02861