Fine-tuning Llama: dataset format and the licence to read first

Published 6 October 2026 · 5 min read

Llama is one of the most fine-tuned open model families, which means the tooling is excellent and the licence is the part people skip. Both matter before you prepare a single row of data.

The licence is not Apache or MIT

This is the first thing to establish, because it shapes what you can do with the result. Llama weights ship under Meta's own community licence rather than a standard open-source licence, with an accompanying acceptable use policy.

Two features matter in practice. There is a scale threshold above which commercial use requires a separate licence from Meta — irrelevant for most organisations and not for all. And the acceptable use policy enumerates prohibited uses that travel with the model, so your deployment has to comply regardless of what your own data permits.

Historically the licence has also carried naming and attribution requirements for derivative models. These terms have changed between releases, so read the licence shipped with the specific checkpoint you are using rather than relying on what was true for an earlier one — and read it before training, not after.

Two scatter plots of target text length against input text length on log scales. In panel a, points are commercial (blue circles), non-commercial or academic (red crosses) and unspecified (yellow triangles), with red crosses concentrated at long target lengths. In panel b, regular datasets (blue) sit at short targets while synthetic ChatGPT, GPT-3 and GPT-4 datasets cluster at long ones. — Fine-tuning datasets plotted by mean prompt length against mean response length (characters, log scale), coloured by licence category (a) and by origin (b). The sets with long responses, the kind chat fine-tuning wants, are mostly non-commercial (red) and synthetic, generated with OpenAI models: their terms can block a Llama fine-tune before the model licence does. Image: Fig. 2 of the article · CC BY 4.0 · from Longpre et al. 2024, Nature Machine Intelligence, “A large-scale audit of dataset licensing and attribution in AI”, doi:10.1038/s42256-024-00878-8

Use the tokenizer's own template

Llama's chat format uses its own special tokens and header structure, and it has changed between major versions. Writing those tokens by hand is how people end up with models that never stop generating or that echo role markers as text.

Let the tokenizer do it. The checkpoint ships with its own template, and applying it is one call. In Unsloth's helpers the named templates are version-specific — there are separate entries for the Llama 3 and 3.1 conventions — which tells you how much the details matter. Our conversation formats post covers why storing rendered tokens in your dataset is a mistake.

Then print the rendered string once and look at it. Thirty seconds here catches a mismatched template, a doubled beginning-of-sequence token, or a missing stop token.

render, then inspect

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("<llama-checkpoint>")

msgs = [
    {"role": "system",    "content": "You are a concise technical assistant."},
    {"role": "user",      "content": "Summarise this changelog entry: ..."},
    {"role": "assistant", "content": "Adds retry handling to the upload path."},
]

text = tok.apply_chat_template(msgs, tokenize=False)
print(repr(text))        # read the special tokens, once, before training

# inference: stop after the assistant header so the model continues it
prompt = tok.apply_chat_template(msgs[:-1], tokenize=False,
                                 add_generation_prompt=True)

A six-stage horizontal flow chart in rounded boxes: collect real examples; create synthetic data; use Llama-3.3-70B-Instruct to create a story or context with factual data from the previous step; create a report from that context using Llama; human verification of the data collected; check for hallucinations. — A synthetic data pipeline with human verification and a hallucination check as named stages rather than afterthoughts. The two steps most synthetic pipelines skip are the two that decide whether the output is usable. Image: meta-llama/llama-cookbook · MIT repository

What the base model already does well

Llama's instruct checkpoints are strong on general instruction-following, English chat, and code to a reasonable standard. Training on data that teaches those again is wasted effort and risks degrading what is already there.

Two line charts titled Continued pretraining and Instruction finetuning, Llama-2-7B trained on Starcoder-Python and on Magicoder-Evol-Instruct-110K: the average of HellaSwag, ARC-challenge and WinoGrande against training tokens or epochs, for LoRA at rank 16, 64 and 256 and for full fine-tuning, with a dashed base-model line at about 0.67; full fine-tuning drops lowest, to about 0.55 and 0.41. — Llama-2-7B's average on HellaSwag, ARC-Challenge and WinoGrande while it is trained on code, by continued pretraining (left) or instruction fine-tuning (right). Every run falls below the base model's score, full fine-tuning furthest. What the base model already does well is at risk from any narrow dataset, so keep some general data in the mix and test for it. Image (cropped): Figure 2 (top row) of the paper · CC BY 4.0 · from “LoRA Learns Less and Forgets Less”, arXiv:2405.09673

So the useful question is what it is not good at for your purposes. Typically: your domain vocabulary, your output format, your refusal boundaries, and languages outside its better-served set. Those are what your data should target.

Test this before collecting anything. Prompt the instruct model on your evaluation set with few-shot examples. If careful prompting gets most of the way there, you need a small behavioural fine-tune rather than a large one — and if it cannot do the task even with help, you may be looking at a knowledge gap, which is continued pretraining rather than instruction tuning. Our what dataset should I use post walks that decision.

A line chart titled 'RAFT LLM_as_judge comparison' of correct percentage against RAG_document_topK at 3, 5 and 7, for all_data_RAFT, pytorch_only_RAFT, llama_only_RAFT, 8B_baseline and 70B_baseline. The 70B baseline runs from 69.01% to 74.65%; all_data_RAFT rises from about 58% to 76.06% at top-k 7, finishing above it; llama_only_RAFT flattens at 64.79%, level with the 8B baseline. — Llama 3 8B Instruct fine-tuned on question-answer data generated from Llama's and PyTorch's documentation, against the untuned 8B and 70B with the same retrieval, scored by a 70B judge. With seven retrieved documents the tuned 8B (76.06%) passes the 70B (74.65%). The run on Llama's own pages alone, about 1,980 examples, ends level with the untuned 8B, which the cookbook puts down to too little data. Image: meta-llama/llama-cookbook · MIT repository

Practical settings

The Llama family is the best-supported target in every fine-tuning framework, so the ecosystem defaults are well tested here. For a LoRA, rank 16 to 32 with alpha equal to the rank or twice it, a learning rate around 2e-4, one to three epochs, and loss masked to assistant turns only.

Memory, from Unsloth's published requirements table: an 8B model needs about 6 GB for 4-bit QLoRA and about 22 GB for 16-bit LoRA; a 70B model needs about 41 GB for QLoRA and about 164 GB for LoRA. That 8B QLoRA figure is what makes this family practical on consumer hardware.

For preference tuning afterwards, drop the learning rate by one to two orders of magnitude — to around 5e-6 or lower — and run one epoch. See the TRL dataset contract for the data shape, and the Unsloth post for the single most common data bug, which is a missing stop token.

During training a LoRA adapter never materialises ΔW: it stores the two thin factors and applies them as two small matrix multiplies. For deployment you keep them separate or merge BA into W once. Rank sets capacity, so a rank too low for the shift you are asking for cannot be fixed with more data or more epochs.

Questions people ask

Can I use a Llama fine-tune commercially?

Generally yes under the community licence, subject to the acceptable use policy and the user-scale threshold above which a separate licence from Meta is required. The terms have changed across releases, so read the licence for your specific checkpoint. Your training data's licence is a separate question that also has to permit the use.

Should I fine-tune the base or the instruct model?

Instruct, for most purposes — it already follows instructions and you are adjusting behaviour. Choose the base model if you are doing continued pretraining on raw domain text, or if you want full control of the response style without inheriting the instruct tuning's habits. Note that fine-tuning an already-tuned model can alter capabilities it had, so evaluate what you are not trying to change.

Which Llama size should I start with?

The smallest one that can do the task with good prompting, because serving cost is forever while training cost is once. Establish the task is learnable on an 8B-class model before reaching for a 70B, and note the memory gap: roughly 6 GB versus 41 GB for QLoRA.

How much data do I need?

The published figures come from Unsloth's documentation, not from the model's maker: a minimum of 100 rows with over 1,000 preferable for supervised fine-tuning, at 1–3 epochs, and at least 500 rows for GRPO. Nothing is published for DPO-style preference tuning. See how much data for a LoRA for the doubling experiment that answers it for your task rather than in general.

Keep reading

Fine-tuningtorchtune has wound down: what to use instead Fine-tuningFine-tuning with Unsloth: what your dataset has to look like Fine-tuningLLaMA-Factory datasets: the dataset_info.json mapping layer