Speech recognition wants varied, noisy, realistic audio. Speech synthesis wants the opposite — and that single inversion explains why most large speech corpora are poor TTS data.
An honest note on availability
Our verified set of 434 open datasets contains a great deal of speech data and only a handful of sets whose documentation describes them as suitable for synthesis specifically. A few large ASR corpora mention TTS as a secondary use.
That reflects the requirement rather than a gap. ASR corpora are built for variety and TTS needs consistency, so most speech data is simply the wrong shape — which is worth knowing before you download three hundred hours and discover it. Our speech and audio datasets and speech recognition datasets guides cover the speech sets that do exist.
So this post is about what the data must look like, and the fact that recording your own is more tractable here than for almost any other task.
The inversion
For recognition you want variety, because the model must be robust to whatever arrives: accents, noise, channels, distances, overlapping speakers. Every bit of messiness in training buys robustness.
For synthesis the model reproduces what it hears. Background noise in training data becomes background noise in generated speech. Inconsistent microphone placement becomes inconsistent timbre. Variable recording levels become variable output. Every bit of messiness in training becomes a defect in the product.
So TTS data wants: a single speaker or cleanly separated multiple speakers, studio or studio-like conditions, a consistent microphone and placement, a high sampling rate, a quiet noise floor, consistent levels, and accurate transcripts.
This is why a large crowdsourced ASR corpus, recorded by thousands of people on their own devices in their own rooms, is excellent recognition data and unusable for synthesis.
What the transcripts must carry
Transcription requirements are stricter than for recognition, and in a different direction.
Exact correspondence. The text must match the audio precisely, including every word actually spoken. A recognition model tolerates some transcript noise; a synthesis model learns a wrong mapping from text to sound, which manifests as mispronunciation.
Normalisation decisions made explicitly. How are numbers, dates, abbreviations, currency and symbols written, and how are they spoken? “£24.50” can be read several ways. Whatever convention your transcripts use is what the model learns, so the normalisation pipeline is part of the product.
Punctuation. It carries prosody — pauses, intonation, phrasing. Unpunctuated transcripts produce flat-sounding speech, which is a common disappointment with data repurposed from recognition corpora.
Pronunciation for the awkward cases. Proper nouns, domain terms, loanwords and initialisms are where synthesis most visibly fails. If your domain has them, a pronunciation lexicon alongside the transcripts is worth the effort.
The consent question is specific here
TTS carries a rights issue that recognition does not, and it has become sharper.
A model trained on one person's voice can reproduce that voice. That is a materially different proposition from recognising speech in general, and voice is treated as personal and in some regimes biometric data. Voice cloning specifically is increasingly regulated.
The practical implication: consent to be recorded for speech recognition research is not consent to have your voice synthesised. Many older corpora predate the distinction entirely, so their consent documentation simply does not address it. A permissive licence does not resolve this, because the licence comes from the compiler and the voice belongs to the speaker.
For commercial synthesis, the defensible positions are a voice you have explicit, specific consent to synthesise — typically a paid voice artist under a contract that says so — or a synthetic voice not modelled on an identifiable person. This is a question to settle before recording, not after training.
Recording your own is unusually practical
Given the consistency requirement, this is one of the few tasks where producing your own data is often easier than finding it.
A single speaker in a treated room with a decent microphone, reading prepared text, produces exactly the consistent high-quality paired data the task wants. Useful results have been achieved with single-digit to low tens of hours for a single voice, and modern approaches adapting a pretrained multi-speaker model need considerably less than training from scratch.
Make the script cover what you need: phonetic coverage so every sound appears in varied contexts, your domain vocabulary and proper nouns, sentence lengths and structures matching your real output, and the prosodic range you want — statements, questions, lists.
And record consistently. One microphone, one position, one room, kept identical from session to session, with levels checked each time. A few days of careful recording beats weeks of searching for data that does not exist in the shape you need.
Questions people ask
Can I use an ASR corpus for TTS?
Usually poorly. Crowdsourced multi-speaker recognition corpora are too varied and too noisy. The partial exception is audiobook-derived corpora such as LibriTTS: long read passages with the book text as transcripts, which is why those sets get mentioned for both uses. But they are volunteer recordings rather than studio sessions, and LibriTTS alone spans 2,456 speakers, so they need filtering by speaker and by recording quality before they make good synthesis data.
How many hours for one voice?
Single-digit to low tens of hours for a good single-speaker voice, and less when adapting a pretrained multi-speaker model rather than training from scratch. Consistency matters more than duration: ten careful hours beats thirty inconsistent ones.
What about multi-speaker and voice cloning?
Multi-speaker models need many speakers with clean separation and speaker labels. Few-shot cloning from seconds of audio is an active area — and the one where the consent question is most acute, since the whole capability is reproducing a specific identifiable person's voice.
Does the text need to be in the target domain?
Partly. Phonetic coverage matters most, and generic text provides it. But domain vocabulary, proper nouns and the sentence structures you will actually synthesise are worth including deliberately — they are where synthesis audibly fails.