Fine-tuning Whisper: audio pairs and transcription conventions

The data shape, and the sampling rate

An ASR dataset is audio paired with its transcription. One field holds the waveform, one holds the text. There is no prompt and no chat template.

The non-negotiable part is sampling rate. Whisper expects audio at a specific rate, and feeding it anything else without resampling produces garbage rather than an error — the model receives a time-distorted signal and transcribes accordingly. Resample on load and assert the rate rather than trusting your source files, because a mixed-rate corpus is extremely common and the failure is silent.

Segment length matters too. The architecture processes fixed-length windows, so very long audio must be chunked and very short clips are padded. If your real inputs are long recordings, train on realistically segmented audio rather than on neat short clips, or the model will not have seen the thing it has to do.

from datasets import load_dataset, Audio

TARGET_SR = 16_000      # what the model expects - check your checkpoint

ds = load_dataset("audiofolder", data_dir="./audio")
ds = ds.cast_column("audio", Audio(sampling_rate=TARGET_SR))   # resample on load

ex = ds["train"][0]
assert ex["audio"]["sampling_rate"] == TARGET_SR, "wrong rate - silent failure ahead"

# each row is just this:
#   {"audio": {"array": <waveform>, "sampling_rate": 16000},
#    "sentence": "the transcript exactly as you want it output"}

Your transcription convention is your output format

This is the decision people make by accident. The model reproduces whatever convention your transcripts use, so each of these is a product choice:

Numbers. “twenty-five” or “25”. Whichever your transcripts use is what you will get, forever.

Punctuation and capitalisation. Present or absent. Many ASR corpora are lowercase and unpunctuated, and a model trained on them produces unpunctuated output that then needs a separate restoration step.

Disfluencies. Whether “um”, false starts and repetitions are transcribed. Verbatim transcripts suit analysis and legal use; cleaned transcripts suit subtitles and notes.

Domain terms. Whether specialist vocabulary is spelled out, abbreviated or expanded.

Mixed conventions are worse than any single choice. A corpus assembled from several sources with different conventions teaches the model to vary unpredictably, which is the hardest failure to fix later. Normalise everything to one convention before training, and write that convention down.

When fine-tuning is worth it

The base model is strong on general speech across many languages, so test it first — on your actual audio, not on a clean sample. If it already works, you are done, and that is the best outcome available.

Where fine-tuning reliably helps: domain vocabulary the model mis-transcribes, such as drug names, part numbers or legal terms; accents and dialects under-represented in pretraining; consistently poor channel conditions such as telephone audio, a specific noisy environment, or far-field microphones; low-resource languages; and enforcing your transcription convention.

Where it helps less: general accuracy on clean speech, which is already good, and anything that is really an audio quality problem. If the recording is unintelligible to a human listener, no fine-tune recovers the words. Our speech recognition datasets post covers choosing public data for the cases where you need to supplement your own.

Practical notes

Hours of audio is the scale unit, and useful fine-tunes have been achieved with single-digit to low tens of hours for a narrow domain or accent. That is a tractable collection job, particularly if you already have recorded calls or meetings with any form of transcript.

If your transcripts came from a previous ASR system, be careful: you will be training the model to reproduce that system's errors. Correct a sample by hand and measure the error rate in your own labels before treating them as ground truth.

Evaluate with word error rate on held-out audio from the same conditions as deployment, and keep a slice of your hardest audio in the evaluation set specifically. Aggregate word error rate hides exactly the failures that matter — one critical mis-transcribed drug name barely moves the average and may be the entire point of the system.

Questions people ask