Axolotl's selling point is that you describe the run in YAML rather than code, and that it will read your data in whatever shape it already is. For irregular datasets that flexibility is the reason to choose it.
The default: a messages list
Axolotl's current default conversation format is a messages list of {role, content} objects — the same shape TRL treats as canonical. The legacy conversations list with {from, value} can still be read by remapping its keys, though the old sharegpt type itself is deprecated.
You declare the dataset and its type in YAML, and Axolotl handles tokenisation and template application. The type: chat_template setting tells it to use the model's own chat template rather than a hardcoded prompt format, which is almost always what you want.
base_model: <model-id>
datasets:
- path: ./data/train.jsonl
type: chat_template
field_messages: messages
message_property_mappings:
role: role
content: content
roles_to_train: ["assistant"] # loss on assistant turns only
adapter: qlora
load_in_4bit: true # required whenever adapter is qlora
lora_r: 32
lora_alpha: 32
sequence_len: 4096
num_epochs: 2
learning_rate: 2e-4
Remapping keys instead of rewriting files
The feature worth knowing about is message_property_mappings. If your data uses speaker and utterance instead of role and content, you say so in the config and train on the files as they are.
This matters more than it sounds. Rewriting a large dataset to satisfy a framework costs disk, time and a conversion step that can introduce errors. Declaring the mapping costs four lines and leaves the original untouched — which also means the same files can feed a different framework later with a different mapping.
roles_to_train is the companion setting and controls loss masking. Set it to ["assistant"] for ordinary instruction tuning; include more roles only if you deliberately want the model to learn to produce them.
Pre-tokenised data
Axolotl will accept data you have already tokenised: input_ids, attention_mask and labels, with masked positions set to -100. You leave the type empty and it trains on what you give it.
This is the escape hatch for anything the template system cannot express — unusual masking, interleaved modalities, a tokenisation scheme you control exactly, or data a vendor ships pre-tokenised. The cost is that your data is now welded to one tokenizer: change the base model and it is worthless. Keep the untokenised source.
{"input_ids": [128000, 9906, ...],
"attention_mask": [1, 1, ...],
"labels": [-100, -100, ..., 9906, 1234, ...]}
# -100 marks positions excluded from the loss.
# Every list in a row must be the same length.
Template-free, with per-span labels
The more distinctive format is input_output, which takes a segments list where each segment carries text and a boolean label.
This gives you span-level control over the loss without any chat template at all. You decide exactly which characters contribute, which is useful for formats that are not conversations: structured documents where only certain fields should be learned, completions where you want loss on a middle section, or anything where the role abstraction does not fit.
It is the most explicit of all the formats and the most work to produce. Reach for it when the conversation model is actively in your way, not as a default.
{"segments": [
{"text": "### Context\nPatient notes: ...\n\n", "label": false},
{"text": "### Diagnosis\n", "label": false},
{"text": "Stage II hypertension, newly diagnosed.", "label": true},
{"text": "<|end_of_text|>", "label": true}
]}
# label:false -> context the model conditions on
# label:true -> tokens it is trained to produceWhen to pick Axolotl over the alternatives
Choose it when your data is irregular, when you need exact control over masking, or when you are training across multiple GPUs or nodes — it has the strongest multi-node story of the common wrappers, with FSDP and DeepSpeed support and context and expert parallelism.
Choose something else if you want maximum single-GPU speed, where Unsloth's kernels are the draw, or if you want a UI and no config file, which is LLaMA-Factory's territory. All three use PEFT for adapters and TRL for preference and RL stages, but they load and template data in their own ways, so the choice is mostly about ergonomics and scale — check that your data shape is supported before migrating.
Questions people ask
Can one config train on several datasets at once?
Yes — datasets is a list, and each entry has its own type and mapping. That is the clean way to combine an instruction set with a smaller domain set without merging files, and it keeps each source's provenance visible in the config.
How do I check the masking is right before a long run?
Train for a handful of steps and inspect a batch: decode the input_ids and print which positions have labels other than -100. You should see context masked and target tokens unmasked. A loss of exactly zero means everything is masked; a suspiciously low loss early can mean nothing is.
Does Axolotl handle preference data?
Yes, for DPO and the related methods, with the same prompt / chosen / rejected contract TRL defines, since TRL is doing the training. The documentation covers the RLHF configurations separately from the SFT dataset formats.
Is pre-tokenised data faster to train on?
It removes tokenisation from the training loop, which helps if that was your bottleneck — usually only for very large datasets or very fast hardware. The real reason to use it is control, not speed, and the cost is that the data only works with the tokenizer that produced it.