torchtune has wound down: what to use instead

Published 21 October 2026 · 4 min read

If you arrived here looking for torchtune's dataset format, the important thing first: the project is no longer actively maintained. Its own README states that development wound down in 2025. This is a migration note rather than a how-to.

The status, plainly

torchtune was Meta's native-PyTorch fine-tuning library — BSD licensed, deliberately dependency-light, built around hackable training recipes for SFT, knowledge distillation, DPO, PPO, GRPO and quantisation-aware training.

Its README now carries a wind-down notice stating that the project is no longer actively maintained, and its last release predates this writing by well over a year. The code remains available and will keep working against the versions it was built for.

That is a reason not to start new work on it. An unmaintained training library drifts out of compatibility with the model architectures, tokenizers and CUDA versions around it, and the drift is silent until something you need does not load.

What it was good at, and where that quality now lives

torchtune's appeal was transparency. The recipes were readable PyTorch rather than a configuration abstraction, which made it the natural choice if you wanted to understand or modify the training loop rather than configure it. It had very few dependencies, which made it pleasant in constrained environments.

Nothing else occupies exactly that niche. The closest in spirit is writing directly against TRL and PEFT yourself — you keep full control of the loop and the dataset handling, at the cost of the convenience torchtune's recipes provided. If what you valued was hackability, that is the honest replacement.

If what you actually valued was that it worked without much ceremony, the wrappers are a better destination.

Two panels joined by an arrow. Prepared: input x and weights W each pass through a FakeQuant block into a Linear layer that outputs y. Converted: x passes through Quant and Dequant blocks and the quantised weights Wq through a Dequant block into the Linear layer. — Quantisation-aware fine-tuning: fake-quantise during training, convert afterwards. Worth checking a framework supports it before you commit a project, because retrofitting it later means retraining. Image: pytorch/torchtune · BSD-3-Clause repository

Choosing a destination

The three actively maintained wrappers differ mainly in ergonomics. All use PEFT for adapters and TRL for preference and RL stages, and all read the common messages and Alpaca shapes, but each has its own data-loading layer. Those shared shapes are what make migration tractable.

If you valued...	Move to	Because
Readable, modifiable training loops	TRL + PEFT directly	no wrapper layer, you own the loop and the dataset handling
Single-GPU speed and low VRAM	Unsloth	Triton kernels and manual backward passes; fits small GPUs
Config-driven runs and irregular data	Axolotl	YAML configs, key remapping, pre-tokenised and template-free formats
Multi-GPU and multi-node training	Axolotl	FSDP, DeepSpeed, context and expert parallelism
No code at all	LLaMA-Factory	CLI plus a web UI over a wide range of model families

What the migration actually involves

Less than you might fear on the data side, and more on the configuration side.

Four labelled stages joined by downward arrows, each beside a code box. "Raw sample dict": {"text": "What's in this image?", "image": "bird.jpg"}. "Message transform": a Message with role="user" and content holding an image entry (a PIL image) and a text entry. "Model transform": {"tokens": [128000, 128006, ...], "mask": [True, True, ...], "encoder_input": {...}}. "Model input": logits = self._model(**batch). — torchtune's dataset pipeline, here for an image-and-text sample: a raw row, a message transform that turns it into role-tagged messages, and a model transform that tokenises and templates them into tokens and a mask. Only the top box is your data; the two transforms are what a migration re-expresses, which is why the files usually move unchanged. Image: pytorch/torchtune · BSD-3-Clause repository · Copyright 2024 Meta

Six prompt templates for the same passage-and-answer task, the original 'Passage: <text> Answer: <text>' and variants with changed separators, casing and spacing, each linked by a dotted arrow to its accuracy on a scale from 0 to 1; the spread runs from 0.036 to 0.804. — The same one-shot classification task given to LLaMA-2-7B with small, meaning-preserving changes to the prompt template, such as a different separator, casing or spacing: accuracy ranges from 0.036 to 0.804. A migration that changes template or masking details needs your evaluation re-run before its numbers are trusted. Image: Figure 1 of the paper · CC BY 4.0 · from Sclar et al. 2023, “Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting”, arXiv:2310.11324

Your datasets are probably fine. torchtune consumed instruction and chat data in the same broad shapes the rest of the ecosystem uses, and the destination frameworks all read a messages list of {role, content} or an Alpaca-style flat row. If your data is in one of those, it transfers unchanged — and Axolotl and LLaMA-Factory can remap unusual field names in configuration rather than requiring a rewrite.

What does not transfer is the recipe configuration: your hyperparameters, masking setup and distributed settings all need re-expressing in the new tool's idiom. Budget for re-validating rather than re-collecting — run your existing evaluation set against a checkpoint trained in the new framework before you trust the numbers, because small differences in template application and loss masking do change results.

A results table from torchtune's knowledge-distillation tutorial. Rows are baseline and LoRA-fine-tuned Llama 3.1 8B and Llama 3.2 1B, then four distillation runs pairing a baseline or fine-tuned 8B teacher with a baseline or fine-tuned 1B student; columns are TruthfulQA mc2, HellaSwag acc and acc_norm, and commonsense acc. The best 1B scores, in bold, fall in different rows for different columns. — torchtune's documented distillation results: the best 1B configuration changes with the benchmark, and the gaps between configurations are mostly a few points or less. Differences of that size are what a template or masking change made during migration could create or erase, so reproduce a table like this on your own evaluation set before trusting the new framework. Image (cropped): pytorch/torchtune · BSD-3-Clause repository

Questions people ask

Is torchtune broken?

No. Unmaintained is not the same as broken — pinned to the dependency versions it was built against, it does what it always did. The risk is forward-looking: new model architectures, tokenizer changes and CUDA updates will not be accommodated, so the set of things you can train with it stops growing.

Can I still read its recipes to learn how fine-tuning works?

Yes, and that is arguably its best remaining use. The recipes are clear, deliberately unabstracted PyTorch, and the mechanics they illustrate — loss masking, distillation, preference optimisation — have not changed. Just do not copy version-specific API calls into new code.

Which replacement has the most similar dataset handling?

TRL directly — writing a formatting function for SFTTrainer is closest to writing your own torchtune transforms, and you keep the same degree of control.

Does this affect PyTorch itself?

Not at all. This is one library in the ecosystem winding down, not a change to PyTorch, which remains the substrate under every alternative listed here.

Keep reading

Fine-tuningFine-tuning with Unsloth: what your dataset has to look like Fine-tuningLLaMA-Factory datasets: the dataset_info.json mapping layer Fine-tuningFine-tuning Llama: dataset format and the licence to read first