Continued pretraining: teaching a model a domain

When you actually need it

Continued pretraining is the right tool in a narrower set of cases than people assume. The test is whether the base model is confidently wrong about facts in your domain, as opposed to knowing the answer and presenting it badly.

Cases where it genuinely helps: a low-resource language the base model barely saw; a technical vocabulary with its own abbreviations and conventions, like clinical notes or a specific legal jurisdiction; an internal corpus of entities and product names that do not exist on the public web; a modality of text the model is weak at, such as heavily tabular or structured documents.

Cases where it does not: the model is too verbose, will not produce valid JSON, ignores your format, refuses too much. All behaviour, all cheaper to fix with supervised fine-tuning.

The data shape, and the mix

Same shape as pretraining — raw documents, one text field, loss on every token. What changes is curation: you are deliberately upsampling a domain rather than sampling the web broadly.

Crucially you do not train on domain text alone. Doing so degrades general capability, and the standard mitigation is to mix a proportion of general-purpose data back in. OLMo 2 illustrates the structure: it anneals its 7B model “three separate times for 50B tokens each” on a curated high-quality mix, with the learning rate decayed linearly to zero over each run.

Order of magnitude for the domain stage: OLMo 2's mid-training runs used 50B to 300B tokens, which it frames as 5–10% of training FLOPs — a small fraction of a stage-one budget, cheap by comparison, and the reason this stage is practical for organisations that could never pretrain.

What annealing buys, quantified

Llama 3 publishes the clearest numbers on this. Annealing on upsampled high-quality code and maths data improved a pretrained 8B model by 24.0% on the GSM8K validation set and 6.4% on MATH.

The same paper describes using annealing as a data-evaluation protocol, which is a useful trick in its own right: take a half-trained 8B model, anneal the learning rate linearly to zero over 40B tokens with 30% weight on the candidate dataset, and see what moves. That is a far cheaper way to assess whether a corpus is worth including than a full training run.

OLMo 2’s totals show how the stages compose: its 7B model is trained on 4.05 trillion tokens of which 3.90 trillion is stage-one pretraining, leaving roughly 150B for mid-training across the three annealing runs.

The embedding learning rate

Here is the detail that trips up people doing this with an adapter rather than a full fine-tune, and it is worth stating plainly because it fails quietly.

Continued pretraining on new domain text often means new vocabulary, so the embedding and output layers need to move — which means including them in the target modules rather than only the attention and MLP projections. Unsloth's documentation for this case adds lm_head and embed_tokens.

But embeddings want a gentler rate than the rest of the network. Unsloth's continued pretraining guidance uses a learning rate of 5e-5 with a separate embedding_learning_rate of 5e-6 — ten times smaller, the top of its advised two-to-ten range — and the separate rate only takes effect through UnslothTrainer with UnslothTrainingArguments; TRL's SFTConfig has no such field. Train the embeddings at the main rate and you can destabilise representations the model already had, producing a model that is worse at everything including your domain.

model = FastLanguageModel.get_peft_model(
    model,
    r = 32,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",
                      "lm_head", "embed_tokens"],   # <- the embeddings, for new vocabulary
    lora_alpha = 32,
    use_gradient_checkpointing = "unsloth",
)

from unsloth import UnslothTrainer, UnslothTrainingArguments

args = UnslothTrainingArguments(      # not SFTConfig: it has no embedding rate
    learning_rate = 5e-5,             # lower than the 2e-4 typical of instruction SFT
    embedding_learning_rate = 5e-6,   # 2-10x smaller again
    ...
)
trainer = UnslothTrainer(model = model, args = args, train_dataset = dataset, ...)

# the dataset is raw text, not instructions:
#   {"text": "<a whole document>"}

Finding domain text at the right scale

The practical obstacle is rarely method and usually supply. Ten billion tokens of a specific domain is a lot of text — far more than most organisations have internally, and the public sources are uneven by field.

Some domains are well served: biomedical literature, legal opinions and filings, patents, code, and the major scientific archives all have substantial open corpora. Others barely exist in public form, and for those the realistic options are your own archives, licensed commercial sources, or synthetic expansion from a smaller real seed.

This is worth scoping before committing to the approach. If you can only assemble a few hundred million tokens of genuine domain text, you are in a different regime — closer to a light domain nudge than to teaching a field — and the honest expectation should be set accordingly.

Questions people ask