Fine-tuning language models, hands-on

SFT, LoRA, QLoRA, continued pretraining, DPO, GRPO and distillation: how each one works, when to use it, and the code to run it.

Fine-tuning takes a pretrained language model and trains it further on a selected dataset. It can teach a response format, improve a domain task, or make tool calls more consistent. This article explains the main methods, their memory costs, the data they need, and how to evaluate the result.

This article draws on the books, papers, public notebooks and library documentation listed below. Its research snapshot is 26 September 2026; the examples and sources were reviewed on 30 September. The training examples have not been executed for this article. They target the TRL 1.14, PEFT 0.21 and Transformers 5.17 API families; pin and test a complete environment before a training run.

How to read it

Parts 1–3 introduce the concepts. Parts 4–5 explain memory requirements and adapters. Parts 6–10 provide training examples, starting with a small encoder and moving through SFT, continued pretraining, DPO and GRPO. Parts 11–12 cover evaluation and common failures. GPU availability and memory requirements depend on the model and configuration.

• essentials
• 1 training
• 2 stages
• 3 when
• 4 memory
• 5 lora
• 6 lab: encoder
• 7 lab: sft
• 8 lab: cpt
• 9 lab: dpo
• 10 lab: grpo
• 11 evaluation
• 12 pitfalls
• sources

The essentials

• Fine-tuning is the pretraining loop, restarted on your data. The model predicts the next token, the error is measured, and every weight moves a small step to reduce it. What the model learns depends on the data you feed it: raw text teaches the language and knowledge of a domain (continued pretraining), prompt and answer pairs teach behavior and format (supervised fine-tuning, SFT), pairs of a better and a worse answer teach preferences (DPO), and prompts a program can grade teach the model to get checkable results right (GRPO).
• Try a good prompt and retrieval first. Fine-tuning improves tone, structure, consistency, domain vocabulary and instruction following. It does not reliably give a model current facts: that is what retrieval (RAG) is for. Rahul's guide puts it as a rule: fine-tune only after a prompt-only baseline fails at something specific.
• The examples matter more than the hyperparameters. A few hundred reviewed examples can be enough for one narrow task: 200 to 500 in Rahul's guide, a few hundred in Iozzia's book, 1,000 in the LIMA paper for a general assistant. Split the data by source (document, company, case), never by row, and keep a test set you look at once.
• LoRA and QLoRA make it fit on one GPU. Full fine-tuning with the Adam optimizer needs about 16 bytes of GPU memory per parameter, 112 GB for a 7B model. LoRA freezes the model and trains small add-on matrices (under 1% of the weights); QLoRA also stores the frozen model in 4 bits, about 3.5 GB of weights for 7B. For SFT, LoRA matches full fine-tuning on small and medium datasets when it is applied to every layer with a learning rate about 10 times higher.
• Continued pretraining changes domain behavior and can erode other skills. The papers below explore replay of general text alongside domain text. Treat the replay fraction and training duration as experimental choices, and monitor both domain and general evaluations.
• Preference tuning and RL come after SFT. DPO trains on chosen and rejected answers with no reward model and a low learning rate. GRPO samples several answers per prompt, scores them with a program and reinforces the ones above the group average. In Raschka's repo, 50 GRPO steps take a 0.6B model from 15.2% to 47.4% on the MATH-500 benchmark.
• Judge the result against the baseline, never by the training loss. Compare the fine-tuned model and the untouched one on the same held-out questions, with a metric of your domain, a regression suite for the general skills it may have lost, and a person reading the errors.

Part 1 · What training does to a model

A language model is a function with millions or billions of numbers inside, called weights or parameters. It reads text split into tokens (pieces of words; Qwen3's vocabulary has 151,936 of them) and outputs a probability for every token in the vocabulary as the next one. Training adjusts the weights so that the real next token gets a higher probability. Pretraining, continued pretraining and SFT all run this same loop; what changes is the data and which tokens count.

The words in every training log

Part 2 · The stages of a model's life

An open model goes through several training stages before you download it, and each one uses a different kind of data. Knowing which stage a method belongs to tells you what data it needs and what it can change.

Base and instruct models

The difference is easy to see. Asked "What is 1 + 1?", a base model continues the text the way a web page would: "2. What is 1 + 1 + 1? 3." (the example is from Hands-On Large Language Models, chapter 12). The instruct version of the same model answers the question and stops. The knowledge was already in the base model; SFT taught it the format of a conversation.

What each stage can change

The LIMA paper (Zhou et al., 2023) fine-tuned a 65B model on only 1,000 curated examples and got answers people rated equal to or better than GPT-4's in 43% of cases. From that it proposed the superficial alignment hypothesis: "A model's knowledge and capabilities are learnt almost entirely during pretraining, while alignment teaches it which subdistribution of formats should be used." Nathan Lambert, in his RLHF book, agrees that the base model sets most of the ceiling, and calls post-training "extracting potential by amplifying valuable behaviors in the base model". He disagrees with the word "superficial": RL can teach a model long chains of reasoning it did not produce before. A practical interpretation: SFT is very good at behavior and format, and weak at adding facts; knowledge comes from pretraining, continued pretraining or retrieval.

The data formats, one line per example

All the labs below read JSONL: a text file with one JSON object per line. These are the four shapes the current libraries accept (TRL's documentation).

# continued pretraining: plain text, loss on every token
{"text": "Artículo 23. Rendimientos del capital inmobiliario ..."}

# SFT, prompt and completion: loss only on the completion
{"prompt": [{"role": "system", "content": "Answer from the context only."},
            {"role": "user", "content": "Context: ...\nQuestion: ..."}],
 "completion": [{"role": "assistant", "content": "..."}]}

# preference tuning (DPO): one prompt, a better and a worse answer
{"prompt": [{"role": "user", "content": "..."}],
 "chosen": [{"role": "assistant", "content": "..."}],
 "rejected": [{"role": "assistant", "content": "..."}]}

# RL (GRPO): only the prompt, plus whatever the reward function needs
{"prompt": [{"role": "user", "content": "What is 17 x 24?"}], "answer": "408"}

Part 3 · Should you fine-tune at all?

The first decision is whether to train anything. There are three approaches, from cheapest to most expensive: write a better prompt; add retrieval (RAG, retrieval-augmented generation: search your documents for the passages relevant to each question and paste them into the prompt); or fine-tune. Rahul's guide asks for three baselines before any training (prompt only, prompt plus retrieval, fine-tuned model) and sets the rule: "Only start fine-tuning after your prompt-only baseline fails at something specific."

What fine-tuning can and cannot fix

• It can fix behavior. In Rahul's list: "tone, structure, consistency, domain vocabulary, instruction compliance". His claim, not backed by an experiment in the article, is that a 1.5B model fine-tuned on 200 to 500 good examples can beat a frontier model with a generic prompt at one narrow task.
• It cannot reliably add current facts. Rahul: for market data or anything that changes, "use retrieval. Not training." Iozzia (chapter 2.5) gives RAG three advantages: up-to-date information, answers you can trace back to their source, and lower cost because there is no training and little labeling.
• It wins on patterns the model doesn't have. Iozzia: fine-tuning "often outperforms RAG on tasks that require learning complex patterns and relationships, such as drug discovery or code generation", and it learns "specialized jargon, concepts, and structures". With LoRA, its cost can be "comparable to, or even lower than" running a RAG system's infrastructure.
• The two combine. Iozzia: "Fine-tuning or PEFT can reduce domain-specific hallucinations, whereas more generic domain hallucinations can be better mitigated with RAG." Rahul's target architecture is a brief, then optional retrieval, then a small fine-tuned model, which is the shape of a RAG assistant whose generator has been fine-tuned to use the context well.

Why small, domain-specific models

Iozzia's book defines small language models (SLMs) by size: they "typically range from a few hundred million to a few billion parameters; there's no official cutoff, but most stay under 10 billion". They are the same technology as large models, only smaller, and the point of making them domain-specific is that "you can specialize them for specific domains and tasks at relatively low cost, using domain-specific and private data". They run on CPUs or consumer GPUs and keep data on your own machines, which is why regulated industries use them. His three options for specializing one:

Condensed from Table 1.1 of Iozzia's book. Catastrophic forgetting is his term for "a noticeable drop in performance on general knowledge or initial capabilities due to a model being trained on new data and tasks". His conclusion: "When you are specializing an SLM to a given domain and task, full training is rarely the best option."

Match the failure to the fix

The following is an experimental planning checklist based on the public SFT, DPO and continued-pretraining methods discussed here. Each row proposes a candidate experiment; it does not establish which method will win on a particular dataset.

Part 4 · Why memory decides the method

Full fine-tuning, LoRA and QLoRA differ mainly in how much GPU memory they need, and memory is what decides which one you can run. It helps to know where it goes.

Where the memory goes

• Weights: 2 bytes per parameter in 16-bit formats (bf16 or fp16).
• Gradients: one number per trainable weight, another 2 bytes.
• Optimizer state: AdamW keeps, for every trainable weight, a 32-bit master copy and two running averages (momentum and variance): 12 bytes.
• Total: 2 + 2 + 12 = 16 bytes per parameter, the figure in the ZeRO paper (Rajbhandari et al., 2019). Hugging Face's memory guide counts 18 because it keeps gradients in 32 bits.
• Activations come on top: the intermediate results of the forward pass, kept for backpropagation. They grow with the batch size and the sequence length.

Arithmetic from the ZeRO paper's 16 bytes per parameter. For comparison, Unsloth's own table of minimum GPU memory (vendor figures, activations included) gives 5 GB for QLoRA and 19 GB for LoRA on a 7B model, 6 and 22 GB on 8B, 8.5 and 33 GB on 14B, and 41 and 164 GB on 70B. The QLoRA paper brought a 65B model from more than 780 GB to under 48 GB.

The other ways to save memory

• Gradient checkpointing. Throws away most activations during the forward pass and recomputes them during backpropagation. Less memory, slower steps. TRL turns it on by default.
• Gradient accumulation. Small batches that add up to a bigger effective batch (Part 1).
• Paged or 8-bit optimizers. Paged optimizers move optimizer state to CPU memory during spikes (QLoRA); 8-bit ones store it in fewer bytes.
• Shorter sequences. max_length caps the tokens per example. Set it from the length of your real examples, not from the model's maximum.
• The right 16-bit format. bf16 is safer for training but the T4 GPU in Colab's free tier doesn't support it, so there you use fp16. The labs pick one automatically.

What fits where

Part 5 · LoRA and QLoRA from the inside

LoRA (low-rank adaptation, Hu et al., 2021) is the method behind almost every fine-tune in this article. It freezes the model and trains a small side path next to each large matrix. QLoRA (Dettmers et al., 2023) is LoRA on a model stored in 4 bits.

What the papers measured

• LoRA (Hu et al., 2021). On GPT-3 175B it cut trainable parameters by 10,000 times and GPU memory by 3 times (1.2 TB to 350 GB), and with rank 4 the saved checkpoint went from 350 GB to 35 MB. Quality held: 73.4 against 73.8 for full fine-tuning on WikiSQL, and 91.7 against 89.5 on MNLI. They adapted only two attention matrices (q and v). Hands-On LLMs works the same example: one 12,288 × 12,288 matrix of GPT-3 is 150 million weights, and rank 8 needs 2 × 12,288 × 8 = 197K (the book prints "12,288 × 2", a typo).
• LoRA Learns Less and Forgets Less (Biderman et al., 2024), on Llama 2 7B. For instruction tuning, a high rank closes the gap: rank 256 scores 0.498 on the HumanEval code benchmark against 0.497 for full fine-tuning, while rank 16 reaches 0.358. On math, rank 64 is enough (0.634 against 0.642 on GSM8K). For continued pretraining LoRA stays behind in every setting (next lab). In exchange it forgets less of what the model knew.
• LoRA Without Regret (Schulman and Thinking Machines Lab, September 2025). LoRA matches full fine-tuning on small and medium SFT datasets and falls behind when the dataset exceeds its capacity. It must go on every layer, especially the MLP (the feed-forward block): attention only at rank 256 loses to MLP only at rank 128. Its best learning rate is consistently about 10 times that of full fine-tuning. It tolerates large batch sizes worse. For RL, even rank 1 matches full fine-tuning.

The hyperparameters

QLoRA: the same on a 4-bit model

The paper's recipe: storage type NF4, compute type bf16.

• NF4 (NormalFloat, 4 bits). A 4-bit number has 16 possible values. Model weights are roughly normally distributed around zero, so NF4 places those 16 values where the weights are dense instead of spacing them evenly. Hands-On LLMs draws it as narrow bins in the middle and wide ones in the tails.
• Blocks and double quantization. Weights are quantized in blocks of 64, each with its own 32-bit scale, which costs 0.5 extra bits per weight. Double quantization stores those scales in 8 bits (in blocks of 256) and brings the cost down to 0.127 bits, about 3 GB saved on a 65B model.
• Paged optimizers. Use NVIDIA's unified memory to move optimizer state to CPU memory during spikes, so a long example doesn't crash the run.
• Result. A 65B model fine-tuned on one 48 GB GPU. Their Guanaco 65B reached 99.3% of ChatGPT's score on the Vicuna benchmark after 24 hours on a single GPU; the 7B version needs 5 GB. Their settings: r = 64, α = 16, learning rate 2e-4 for 7B and 13B, 1e-4 for 33B and 65B.

Adapters in production

• Merge when one model serves one task. Merge onto a 16-bit copy of the base model, not the 4-bit one: Hands-On LLMs reloads the base in 16 bits for this, and PEFT warns that merging into 4-bit layers changes the outputs through rounding.
• Keep the adapter separate when one base serves several tasks or clients. vLLM serves several adapters on one base model and picks one per request. Its default maximum rank is 16, so an adapter with r = 32 or more needs --max-lora-rank raised.

Part 6 · Lab 1: fine-tune a small encoder

This lab builds on the encoder fine-tuning approach in chapter 11 of Hands-On Large Language Models. The public companion notebooks provide the source examples. The snippets below add validation, error analysis and alternative training choices. Results attributed to the book are its reported results; the adapted snippets have not been run here.

It is a good first run because it is small and fast. An encoder such as BERT reads a text and produces a label; it can't write text. The full fine-tune below took 61.7 seconds on a free T4 in the book's official notebook, and every idea carries over to the generative labs: the trainer, the learning rate, freezing against LoRA, and adapting to a domain with unlabeled text.

The task: classify movie reviews as positive or negative, with the Rotten Tomatoes dataset (8,530 training, 1,066 validation and 1,066 test reviews) and bert-base-cased.

Full fine-tuning, the baseline

import evaluate, numpy as np
from datasets import load_dataset
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
                          DataCollatorWithPadding, TrainingArguments, Trainer)

data = load_dataset("cornell-movie-review-data/rotten_tomatoes")
id2label = {0: "neg", 1: "pos"}
label2id = {v: k for k, v in id2label.items()}

model_id = "bert-base-cased"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(
    model_id, num_labels=2, id2label=id2label, label2id=label2id)

tokenized = data.map(lambda b: tokenizer(b["text"], truncation=True), batched=True)
f1 = evaluate.load("f1")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    return f1.compute(predictions=np.argmax(logits, axis=-1), references=labels)

args = TrainingArguments(
    output_dir="outputs/full-finetune-bert",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=1,
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="f1",
    report_to="none",
    seed=42,
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=tokenized["train"],
    eval_dataset=tokenized["validation"],       # choose runs on validation
    processing_class=tokenizer,
    data_collator=DataCollatorWithPadding(tokenizer),  # pads each batch to its longest text
    compute_metrics=compute_metrics,
)
trainer.train()
trainer.evaluate(tokenized["test"], metric_key_prefix="test")  # once, at the end

This configuration illustrates a classification run. The metric function can also report accuracy, precision and recall.

Train less of the model

Next, compare how much of the model needs to move. Freezing means setting requires_grad = False on a weight, so it gets no gradient and never changes:

def freeze_bert_lower_layers(model, freeze_through_layer=9):
    """Freeze the embeddings and encoder blocks 0..9; train 10, 11 and the head."""
    for name, param in model.named_parameters():
        if name.startswith("bert.embeddings"):
            param.requires_grad = False
        elif "bert.encoder.layer." in name:
            layer_idx = int(name.split("bert.encoder.layer.")[1].split(".")[0])
            param.requires_grad = layer_idx > freeze_through_layer
        else:
            param.requires_grad = True   # pooler and classification head

And the 2026 version of the same idea, LoRA with PEFT:

from peft import LoraConfig, get_peft_model, TaskType

lora_config = LoraConfig(
    task_type=TaskType.SEQ_CLS,
    r=8,
    lora_alpha=16,
    lora_dropout=0.1,
    target_modules=["query", "value"],   # BERT's names for the q and v matrices
    modules_to_save=["classifier"],      # the new head trains in full
)
lora_model = get_peft_model(base_model, lora_config)
lora_model.print_trainable_parameters()
# same Trainer as above, with learning_rate=1e-3

Continued pretraining for an encoder

Masked language modeling is an encoder objective for continued pretraining. Masked language modeling hides 15% of the tokens and trains the model to fill them in; run on unlabeled text from your domain, it shifts the model's vocabulary toward that domain before you train the classifier.

from transformers import AutoModelForMaskedLM, DataCollatorForLanguageModeling

mlm_model = AutoModelForMaskedLM.from_pretrained("bert-base-cased")
text_only = tokenized.remove_columns("label")          # MLM needs no labels
collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15)
# Trainer(model=mlm_model, data_collator=collator, ...).train()
# save it, then load it with AutoModelForSequenceClassification for the classifier

What the book measured

From chapter 11 of Hands-On LLMs; the timings are from its official notebook. In the book's sweep, training five of the twelve blocks already reached 0.843 against 0.854 for all of them. For the MLM run, the book shows only a qualitative check: asked to fill "What a horrible [MASK]!", plain BERT proposes idea, dream, thing, day, thought; after continued pretraining on the reviews it proposes movie, film, mess, comedy, story. It never measures the classifier trained on top.

Choosing the first experiment

Begin with a small training subset to check the data, loss and evaluation pipeline. Then compare full fine-tuning, head-only training, top blocks and LoRA. For each run, record trainable parameters, validation and test F1, time and GPU memory. As a separate experiment, compare a classifier after domain MLM with the unchanged encoder baseline.

Part 7 · Lab 2: SFT with QLoRA on a small chat model

This lab fine-tunes a small chat model to answer questions from a given context in a fixed format, and to say so when the context doesn't contain the answer. The base model is Qwen/Qwen2.5-1.5B-Instruct, the one Rahul's guide recommends starting with because it is "small enough for QLoRA on a T4 GPU". The code is the same for a newer or larger model; only the chat template needs checking.

Step 1: the baseline

Before any training, write the system prompt you would use in production and run the untouched model over 30 or so real questions. Save the answers. This is what the fine-tuned model has to beat, and the comparison code in step 6 produces both columns at once.

Step 2: the data

• How many. Start with 200 to 500 reviewed examples. Rahul's versioning plan: v0 prompt only, v1 200 reviewed examples, v2 1,000, v3 a larger set plus retrieval.
• What they contain. Real questions, the context chunks your retriever returns for them, and the answer you want. Add examples where the context doesn't contain the answer and the right output is an honest non-answer, balanced with the positive ones.
• Rules from Rahul's guide. Check you have the right to train on each source ("Public availability does not mean training permission"). A person reviews every answer ("Machine draft ≠ ground truth"). Never let a model fix a number: if the source doesn't support a figure, the figure doesn't go in.
• Split by source. Put all the examples from one document (or company, or case) in the same split. If one document's questions land in both train and test, the model is tested on what it memorized, which Rahul calls "the data mistake that makes evaluation meaningless".

Two lines of all.jsonl, from a made-up HR policy, one answerable and one not. The source_doc field is only used for splitting.

{"prompt": [{"role": "system", "content": "Responde solo con el contexto. Si no está, dilo."},
            {"role": "user", "content": "Contexto: Las vacaciones se solicitan en el portal de RR. HH. con 15 días de antelación.\n\nPregunta: ¿Con cuánta antelación pido las vacaciones?"}],
 "completion": [{"role": "assistant", "content": "Con 15 días de antelación, en el portal de RR. HH."}],
 "source_doc": "politica-vacaciones"}
{"prompt": [{"role": "system", "content": "Responde solo con el contexto. Si no está, dilo."},
            {"role": "user", "content": "Contexto: Las vacaciones se solicitan en el portal de RR. HH. con 15 días de antelación.\n\nPregunta: ¿Cuántos días de vacaciones tengo al año?"}],
 "completion": [{"role": "assistant", "content": "El contexto no dice cuántos días de vacaciones corresponden. Consúltalo con RR. HH."}],
 "source_doc": "politica-vacaciones"}

import json, random
from collections import defaultdict

rows = [json.loads(line) for line in open("data/all.jsonl", encoding="utf-8")]
by_doc = defaultdict(list)
for r in rows:
    by_doc[r["source_doc"]].append(r)

docs = sorted(by_doc)
random.Random(42).shuffle(docs)
cut1, cut2 = int(len(docs) * 0.8), int(len(docs) * 0.9)
splits = {"train": docs[:cut1], "validation": docs[cut1:cut2], "test": docs[cut2:]}

for name, names in splits.items():
    with open(f"data/{name}.jsonl", "w", encoding="utf-8") as f:
        for d in names:
            for r in by_doc[d]:
                keep = {"prompt": r["prompt"], "completion": r["completion"]}
                f.write(json.dumps(keep, ensure_ascii=False) + "\n")

Step 3: which tokens the model learns from

Step 4: train

First measure your examples, so max_length fits them instead of the model's maximum:

from transformers import AutoTokenizer
from datasets import load_dataset

BASE = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(BASE)
train = load_dataset("json", data_files="data/train.jsonl", split="train")
lengths = sorted(
    len(tok(tok.apply_chat_template(r["prompt"] + r["completion"], tokenize=False))["input_ids"])
    for r in train
)
print("95th percentile:", lengths[int(len(lengths) * 0.95)], "tokens")

Then the training script. It follows TRL 1.14's documentation and Rahul's settings, with the step and checkpoint numbers sized for about 300 examples:

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

BASE = "Qwen/Qwen2.5-1.5B-Instruct"
use_bf16 = torch.cuda.get_device_capability()[0] >= 8   # Ampere or newer; a T4 is 7.5
dtype = torch.bfloat16 if use_bf16 else torch.float16

data = load_dataset("json", data_files={
    "train": "data/train.jsonl",
    "validation": "data/validation.jsonl",
})

quant = BitsAndBytesConfig(            # the Q in QLoRA: store the base in 4 bits
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=dtype,
)

lora = LoraConfig(                     # the LoRA: small trainable matrices
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    task_type="CAUSAL_LM",
)

args = SFTConfig(
    output_dir="runs/qlora-v1",
    model_init_kwargs={"dtype": dtype},   # TRL loads in float32 if you don't say
    max_length=1024,                      # from the 95th percentile above
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,        # 16 examples per update
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_steps=5,
    eval_strategy="steps",
    eval_steps=10,
    save_strategy="steps",
    save_steps=10,
    save_total_limit=3,
    load_best_model_at_end=True,          # keep the lowest validation loss
    metric_for_best_model="eval_loss",
    logging_steps=5,
    bf16=use_bf16,
    fp16=not use_bf16,
)

trainer = SFTTrainer(
    model=BASE,
    args=args,
    train_dataset=data["train"],
    eval_dataset=data["validation"],
    quantization_config=quant,
    peft_config=lora,
)
trainer.train()
trainer.save_model("runs/qlora-v1/best")   # writes only the adapter, a few MB

• Loss masking needs nothing in the code: with prompt and completion data, TRL computes the loss on the completion only.
• Packing (concatenating several short examples into one sequence) stays off. TRL's documentation says it is meant for setups with FlashAttention; without it, examples can see each other.
• Names changed. Older tutorials, including the two books, pass max_seq_length, tokenizer= or torch_dtype; the current names are max_length, processing_class= and dtype (Part 12 lists the changes).

Step 5: read the curves

Step 6: compare with the baseline

Load the base model in 16 bits, put the adapter on top, and answer every test question twice: with the adapter switched off (the baseline) and on.

import json, torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained(BASE)
base = AutoModelForCausalLM.from_pretrained(BASE, dtype=dtype, device_map="auto")
model = PeftModel.from_pretrained(base, "runs/qlora-v1/best").eval()

def answer(messages):
    inputs = tok.apply_chat_template(messages, add_generation_prompt=True,
                                     return_tensors="pt").to(model.device)
    out = model.generate(**inputs, max_new_tokens=300, do_sample=False)
    return tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

results = []
for line in open("data/test.jsonl", encoding="utf-8"):
    r = json.loads(line)
    with model.disable_adapter():
        before = answer(r["prompt"])
    results.append({"question": r["prompt"][-1]["content"],
                    "expected": r["completion"][0]["content"],
                    "base": before,
                    "tuned": answer(r["prompt"])})

apply_chat_template returns a dictionary of tensors by default since transformers 5, which is why the code passes **inputs to generate. Part 11 turns results into a table a person can review.

Step 7: merge and serve

merged = model.merge_and_unload()     # returns a new model: reassign it
merged.save_pretrained("runs/qlora-v1/merged")
tok.save_pretrained("runs/qlora-v1/merged")

# or serve the base model with the adapter on top (vLLM, OpenAI-compatible API)
vllm serve Qwen/Qwen2.5-1.5B-Instruct --enable-lora \
  --lora-modules faq=runs/qlora-v1/best --max-lora-rank 16
# requests use "model": "faq"

On your Mac: MLX-LM

bitsandbytes, the library behind the 4-bit loading above, was built for NVIDIA GPUs, and its Apple Silicon support is recent (the transformers documentation lists 4-bit loading on macOS 14 or later). On a Mac the simplest route is Apple's MLX-LM, which fine-tunes with LoRA from the command line and reads the same kind of JSONL. It wants the files named train.jsonl and valid.jsonl and, for chat data, the messages format (join the prompt and completion lists into one).

pip install "mlx-lm[train]"
mlx_lm.lora --model Qwen/Qwen2.5-1.5B-Instruct --train --data data \
  --mask-prompt --iters 600          # --mask-prompt: loss on the answer only
mlx_lm.generate --model Qwen/Qwen2.5-1.5B-Instruct --adapter-path adapters \
  --prompt "..."
mlx_lm.fuse --model Qwen/Qwen2.5-1.5B-Instruct --adapter-path adapters

MLX-LM's defaults: LoRA on 16 layers, rank 8, learning rate 1e-5, batch size 4. A model quantized with mlx_lm.convert -q turns it into QLoRA; --fine-tune-type full trains every weight. Unsloth, the other popular option, advertises "2× faster with 70% less VRAM with no accuracy loss" (vendor claim); its Python library runs on Linux and Windows, and its own documentation contradicts itself about training on a Mac.

Starting points

Part 8 · Lab 3: continued pretraining on raw domain text

Continued pretraining (CPT, also called domain-adaptive pretraining) keeps training a model on raw text from your domain with the pretraining objective: predict the next token, loss on every token. Use it when the model doesn't know the language or the facts of the domain and you have far more raw text than question-answer pairs. It is the method with the least coverage in your reading list (Iozzia's book doesn't cover it), so this part rests on three papers and on the encoder version in Lab 1.

What the papers found

• It helps, even on little text (Gururangan et al., 2020, "Don't Stop Pretraining"). They continued pretraining RoBERTa on a large domain corpus (DAPT, domain-adaptive) or only on the unlabeled text of the task's own training set (TAPT, task-adaptive), then fine-tuned. Both improved on plain RoBERTa in the tasks below, and doing both was best in two of the three.
• It makes the model forget, and three habits fix most of it (Ibrahim et al., 2024). Raise the learning rate again at the start (re-warming), let it decay again (re-decaying), and mix in a share of general text like the one the model was pretrained on (replay). With those three, continuing matched retraining from scratch on all the data. "Even the lowest tested replay of 1% significantly reduces forgetting"; they used 5% for a small shift (one English corpus to another) and 25% for a large one (English to German). A higher learning rate adapts more and forgets more.
• LoRA learns less here (Biderman et al., 2024). In continued pretraining on code, the best LoRA reached 0.224 on HumanEval after 20 billion tokens, about what full fine-tuning reached after 4 billion (0.218); full fine-tuning peaked at 0.263. On math text: 0.203 against 0.293 on GSM8K. Use full fine-tuning when the model fits; LoRA forgets less if that matters more.

Gururangan et al. (2020), Table 5, and Ibrahim et al. (2024), Table 2.

The 75/25 split is an illustrative configuration. Ibrahim and colleagues study replay of general data during continued pretraining. Compare candidate mixes using both domain and general held-out measurements.

The order: continued pretraining on the base model, then SFT

Raw text has no conversation format, so continuing to pretrain an instruct model can wear away the chat behavior its SFT gave it. The usual pipeline (Part 2) runs continued pretraining on the base model and SFT afterwards.

How much text

For scale, an illustrative corpus of one million tokens contains roughly 488 full blocks of 2,048 tokens. With 32 blocks per update, that is about 15 full updates per epoch before implementation details such as padding and dropped batches. Repeating a small corpus increases exposure, not information diversity. Compare retrieval, SFT and task-adaptive pretraining on held-out examples rather than choosing from corpus size alone.

The code

from datasets import load_dataset, interleave_datasets

# one document per line: {"text": "..."}
domain = load_dataset("json", data_files="corpus/domain_train.jsonl", split="train")
general = load_dataset("json", data_files="corpus/general_sample.jsonl", split="train")
mixed = interleave_datasets([domain, general], probabilities=[0.75, 0.25], seed=42)

eval_sets = {
    "domain": load_dataset("json", data_files="corpus/domain_heldout.jsonl", split="train"),
    "general": load_dataset("json", data_files="corpus/general_heldout.jsonl", split="train"),
}

import torch
from trl import SFTConfig, SFTTrainer

args = SFTConfig(
    output_dir="runs/cpt-v1",
    model_init_kwargs={"dtype": torch.bfloat16},
    packing=True,
    packing_strategy="wrapped",     # join documents and cut fixed-length blocks
    max_length=2048,
    num_train_epochs=1,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=5e-5,             # a starting point to sweep
    lr_scheduler_type="cosine",
    warmup_steps=100,               # re-warming
    eval_strategy="steps",
    eval_steps=200,
    save_steps=200,
    logging_steps=20,
    bf16=True,
)

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",      # the base model, not the instruct one
    args=args,
    train_dataset=mixed,
    eval_dataset=eval_sets,         # logs eval_domain_loss and eval_general_loss
)
trainer.train()

• Full fine-tuning. There is no peft_config: every weight trains. A 0.5B model needs about 8 GB for weights, gradients and optimizer, so one 24 GB GPU is enough.
• Loss on every token. With a text column, TRL computes the loss over the whole text and adds an end-of-sequence token to each document.
• Packing. wrapped concatenates documents and cuts them into blocks, the classic way to prepare pretraining text. TRL warns that it mixes unrelated examples; that matters for SFT, not for raw text.
• Two validation sets. The domain loss should fall. The general loss should not rise much: if it does, the model is forgetting, and you add replay or lower the learning rate. The exponential of a loss is its perplexity, which you will see reported instead.
• Learning rate. 5e-5 is a starting point, not a finding; Biderman swept from 1e-5 to 5e-4. Higher adapts faster and forgets more.
• With LoRA instead, add a peft_config with a high rank on all linear layers (Biderman used up to 256) and expect it to learn less.

The tokenizer doesn't change. If the domain has words the tokenizer splits into many pieces (Basque, chemical names), continued pretraining can't fix that. Adding tokens to the vocabulary is a separate step that needs new embeddings trained (modules_to_save).

Part 9 · Lab 4: preference tuning with DPO

Preference tuning teaches the model which of two acceptable answers to prefer. The original method, RLHF, has three stages: collect human preferences between pairs of answers, train a reward model (a copy of the model with a head that outputs a single score), and optimize the model against that score with the PPO algorithm. InstructGPT used about 10,000 examples for SFT, 33,000 prompts for the reward model and 31,000 for the RL stage (Lambert). DPO (direct preference optimization, Rafailov et al., 2023) gets the same effect from the same data "without explicit reward modeling or reinforcement learning", with "a simple binary cross entropy objective".

The loss in words

For each pair, measure how much more likely the model in training makes the chosen answer than the reference model did, and the same for the rejected answer. The loss falls when the chosen answer's gain exceeds the rejected one's (Lambert's explanation). beta sets how far the model may move from the reference: a higher value keeps it closer. The DPO paper, Hands-On LLMs and TRL all use 0.1. In code, from the original implementation as quoted by Lambert:

pi_logratios = policy_chosen_logps - policy_rejected_logps
ref_logratios = reference_chosen_logps - reference_rejected_logps
losses = -F.logsigmoid(beta * (pi_logratios - ref_logratios))

The paper's subtitle is "Your Language Model is Secretly a Reward Model": the model's log-probability ratio against the reference acts as the reward that RLHF would have trained separately.

The code

Pairs for the abstention case: the context doesn't contain the answer, the chosen reply says so, the rejected one makes up a figure.

{"prompt": [{"role": "user", "content": "Contexto: Las vacaciones se solicitan en el portal de RR. HH. con 15 días de antelación.\n\nPregunta: ¿Cuántos días de vacaciones tengo al año?"}],
 "chosen": [{"role": "assistant", "content": "El contexto no dice cuántos días corresponden. Consúltalo con RR. HH."}],
 "rejected": [{"role": "assistant", "content": "Tienes 23 días laborables al año."}]}

import torch
from datasets import load_dataset
from peft import LoraConfig
from trl import DPOConfig, DPOTrainer

pairs = load_dataset("json", data_files="data/pairs.jsonl", split="train")

args = DPOConfig(
    output_dir="runs/dpo-v1",
    model_init_kwargs={"dtype": torch.bfloat16},
    beta=0.1,
    learning_rate=5e-6,          # TRL's value for LoRA; 5e-7 for full fine-tuning
    num_train_epochs=1,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    max_length=1024,
    bf16=True,
)

trainer = DPOTrainer(
    model="runs/qlora-v1/merged",   # start from the SFT model of Lab 2
    args=args,
    train_dataset=pairs,
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                           task_type="CAUSAL_LM"),
)
trainer.train()

With a LoRA adapter there is no second copy of the model in memory: TRL switches the adapter off to get the reference model's probabilities.

Where the pairs come from

• People. Labelers pick the better of two answers. Lambert puts a human preference at about $1 or more, sometimes above $10 per prompt, against under a cent with a frontier model as the judge, and advises: "For those getting started, AI feedback should be the first attempt."
• Two models of different strength. Olmo 3 and SmolLM 3 took the chosen answers from Qwen3 32B and the rejected ones from Qwen3 0.6B ("delta learning").
• Your own model's failures. Lambert stresses that answers generated by the model you are training ("on-policy") matter. The wrong answers from your evaluation, paired with corrected ones, are that kind of data.

What to watch

• Low learning rates. DPO only started to work well once people found "surprisingly low learning rates" (Lambert).
• Displacement. The model can lower the probability of both answers, the rejected one more; the loss looks good while the chosen answer also becomes less likely.
• Data first. Lambert: "Change the data before changing the loss." The variants (IPO, SimPO, ORPO, which merges SFT and preference tuning into one run) matter less than the starting model and the data.
• Check it did something. In the official Hands-On LLMs notebook, the DPO model's answer to the test prompt is identical, word for word, to the SFT model's.

Part 10 · Lab 5: reinforcement learning with GRPO, and distillation

Reinforcement learning with verifiable rewards trains on prompts whose answers a program can check, and needs no written answers at all. In Cameron Wolfe's vocabulary: the policy is the model, the reward is the score a program gives an answer, and the advantage is how much better an answer did than expected. PPO estimates "expected" with a critic, a second network trained alongside the model. GRPO (group relative policy optimization) uses the average of a group of answers to the same prompt instead.

Why the critic matters

Wolfe's count: PPO handles up to four models (the policy, the frozen reference, the critic and the reward model), two of them trained. Hosting a model in 16 bits takes about 2 GB per billion parameters and training one about 16 GB per billion. Dropping a trained model saves far more than dropping one that only runs inference, and GRPO drops the critic.

The code

import re
from datasets import load_dataset
from peft import LoraConfig
from trl import GRPOConfig, GRPOTrainer

# {"prompt": [{"role": "user", "content": "What is 17 x 24? Put the result in \\boxed{}."}], "answer": "408"}
dataset = load_dataset("json", data_files="data/grpo_prompts.jsonl", split="train")

def boxed_answer_reward(completions, answer, **kwargs):
    """1.0 when the text inside \\boxed{...} equals the expected answer, else 0.0."""
    rewards = []
    for completion, expected in zip(completions, answer):
        text = completion[0]["content"]            # conversational data: one message
        match = re.search(r"\\boxed\{([^{}]*)\}", text)
        rewards.append(1.0 if match and match.group(1).strip() == expected else 0.0)
    return rewards

args = GRPOConfig(
    output_dir="runs/grpo-v1",
    num_generations=8,              # answers sampled per prompt
    max_completion_length=512,
    per_device_train_batch_size=8,  # the effective batch must divide by num_generations
    gradient_accumulation_steps=4,
    learning_rate=1e-5,             # TRL's value for LoRA; 1e-6 for full fine-tuning
    logging_steps=5,
    bf16=True,
)

trainer = GRPOTrainer(
    model="runs/qlora-v1/merged",
    reward_funcs=boxed_answer_reward,
    args=args,
    train_dataset=dataset,
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                           task_type="CAUSAL_LM"),
)
trainer.train()

The reward function receives every dataset column other than prompt (here answer) as a list aligned with the completions, and returns one number per completion. Several functions can be passed as a list; their rewards are added. TRL's default beta is 0.0: no KL penalty, so no reference model is loaded.

What Raschka measured

From the companion repo of Build a Reasoning Model (From Scratch), chapters 6 and 7: one problem per step, 8 answers per problem, 512 new tokens at most, temperature 0.8, learning rate 1e-5, about 20 GB of memory. Chapter 7's conclusion: "the common recommendation is not to use a KL term when training on math data".

Rules from the sources

• Pick prompts the model sometimes gets right. Lambert: keep problems it solves "only 20-80% of the time", measured by sampling about 16 answers each. Always right or always wrong teaches nothing.
• Watch four numbers (Raschka, chapter 7): the average reward (at 1.0 there is nothing left to learn), the spread of the advantages, the entropy (how varied the answers are; near zero means the model has stopped exploring) and the answer length.
• Keep format rewards small. In Raschka's run a format reward at full weight helped at first and made results worse after about 100 steps. His fixes: weight it around 0.1 instead of 1.0, or give it only when the answer is also correct.
• Many passes over few prompts are normal. Lambert: instruction tuning runs 1 or 2 epochs, RL "hundreds or thousands of epochs over the same few data points".
• Iozzia's chapter 15 runs GRPO on Qwen2.5-3B-Instruct on one 16 GB GPU with Unsloth and LoRA rank 64, and says "we can often get good performance with only 200 to 300 samples". Before reusing its notebook, check the reward functions: the regular expressions look for a literal backslash followed by "n" instead of a line break, and an else hangs from the for loop, so the format rewards can't score as intended.

Distillation

Distillation trains a small model (the student) to copy a large one (the teacher). In hard distillation the teacher writes answers and the student is fine-tuned on them; Raschka: "Technically, this is standard supervised fine-tuning on synthetic data." In soft distillation the student learns to match the teacher's probability for every token, which needs the teacher's raw outputs and the same tokenizer. Hard distillation is the common one.

• The teacher matters more than the epochs. After three epochs the same student scored about 11 points more with the other teacher, and 50 GRPO steps (47.4%) beat both.
• On-policy distillation. Lambert: a student trained only on the teacher's text never practices recovering from its own mistakes (exposure bias). In on-policy distillation the student writes and the teacher scores each of its tokens.
• Healing is this idea after compression: the compressed model is retrained, with the original as teacher or with pretraining and SFT data, to recover what compression cost it.
• Terms of use. Raschka: "even generating and reusing the text outputs for distillation may violate certain providers' policies." Check the teacher's license before distilling from it.

Part 11 · Evaluation: how to know it worked

A falling loss says the training ran; it doesn't say the model is better at your task. Every source treats this as the hard part. Hands-On LLMs: "We want to emphasize the current lack of golden standards. No one metric is perfect for all use cases."

• Compare against the baseline on the same held-out questions. Rahul's list of what to measure for a text-generation task: format compliance, usefulness rated 1 to 10 by a person, the rate of claims the source doesn't support, whether numbers survive intact, repetition, latency and cost per answer.
• Measure what your domain needs. Iozzia's GPT-2 model generates code for the Manim animation library, and he checks it in four steps: does it parse, does it use the library's API correctly (walking the syntax tree), does it run (in a subprocess with a timeout and limited resources), and does the rendered result match a reference. He adds that "you'll still need a qualitative review by subject matter experts".
• Use a model as a judge, and check the judge. An LLM scoring answers scales well and "grows with the field" (Hands-On LLMs), but it has to agree with people first. Compare its ratings with human-reviewed examples from the target task.
• Test what the model may have lost. Keep a fixed regression set covering general tasks and any instruction-following or tool-calling behavior the application depends on. For continued pretraining, monitor general held-out loss alongside domain loss.
• Be wary of public benchmarks. Models can overfit to them and they rarely cover your case. Hands-On LLMs quotes Goodhart's law ("When a measure becomes a target, it ceases to be a good measure") and concludes "you are the best evaluator".
• Read the errors. From the notebook: "A score tells you how much the model is wrong. Error analysis tells you why." Look at the wrong answers before touching a hyperparameter; they often show label noise, ambiguous questions or a metric that doesn't match the need.

A review table in a few lines

This turns the results list from Lab 2 into a CSV a person can go through, with Rahul's check for numbers the model made up: any number in the answer that doesn't appear in the prompt, which holds the context.

import csv, re

def numbers(text):
    return set(re.findall(r"\d+(?:[.,]\d+)?", text))

with open("eval/compare.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(["question", "expected", "base", "tuned",
                     "invented numbers (base)", "invented numbers (tuned)", "better?"])
    for r in results:                       # from Lab 2, step 6
        source = r["question"]              # the prompt carries the context
        writer.writerow([
            r["question"], r["expected"], r["base"], r["tuned"],
            " ".join(sorted(numbers(r["base"]) - numbers(source))),
            " ".join(sorted(numbers(r["tuned"]) - numbers(source))),
            "",                             # a person fills this column
        ])

An evaluation checklist

Part 12 · What usually goes wrong

What changed since the books were written

Both books and many tutorials use names the libraries have since renamed or removed. If a snippet fails, check this table first.

Errors in the sources, so they don't confuse you

• Hands-On LLMs. "Two 12,288 × 2 matrices" should be × 8. It recommends lora_alpha at twice r and then uses r = 64 with alpha = 32. SetFit's "20 × 32 = 680" is 640. The continued-pretraining text says 20 epochs and the code runs 10. The DPO text describes a different TinyLlama from the one the code loads. Its NER metric scores each token as a separate sequence; entity-level evaluation should preserve complete sequences.
• Iozzia. Chapter 3 says it selects runs by "evaluation accuracy" but the code uses the validation loss, and the book runs 10 search trials where the notebook runs 3. Chapter 2 imports prepare_model_for_int8_training and calls prepare_model_for_kbit_training. Chapter 15 trains 200 steps on 200 samples in the book and 10 steps on 100 in the notebook, and its reward functions have the bugs described in Part 10. None of his fine-tuning examples reports a measured before-and-after result.
• Lambert. The Olmo 3 RL run is "4 weeks on 200 GPUs" in chapter 1 and "nearly 4-week-long … 220 GPUs" in chapter 7. The web edition has three sections newer than the print copy (the multi-teacher distillation recipes, the expanded on-policy distillation section and agentic evaluation).
• Rahul. His "example scorecard", the $1.70 cost estimate and the version record are illustrations of a format, not results of an experiment.

Sources

The source research was assembled on 26 September 2026. Library release checks on 30 September found TRL 1.14.1, PEFT 0.21.1 and Transformers 5.17.0. The training code remains unexecuted; benchmark values attributed to a book or paper are reported results from that source.