Speaker identification and diarisation: sessions, not speakers

Published 12 August 2026 · 5 min read

Speaker tasks need a dataset property most speech corpora do not optimise for: repeated recordings of the same person on different occasions. A corpus with thousands of speakers and one recording each is nearly useless here, however large.

An honest note on availability

Searching our verified set of 434 open datasets for speaker-focused sets turned up very few — one canonical speaker-recognition corpus and a dialect-annotated phonetic set, plus a number of ASR corpora that mention speakers incidentally.

We could have padded a list with those ASR corpora. We have not, because a recognition corpus that happens to record speaker identity is generally poor speaker-task data for the reason below, and you would have found that out after downloading it.

Our speech and audio datasets post covers the speech sets in Dataset Finder. This post covers what the task needs, so you can assess a candidate yourself.

Sessions per speaker is the property that matters. One recording each confounds speaker identity with channel, and the metric collapses in deployment.

Three tasks, different data

Worth separating, because they are often discussed together and need different things.

Verification asks whether two recordings are the same person — an open-set, one-to-one comparison. This is what voice authentication does, and it needs the session property described below.

Identification picks a speaker from a known enrolled set — closed-set, one-to-many. Needs enrolment audio per speaker, and gets harder as the enrolled set grows.

Diarisation segments a recording by who is speaking when, usually without knowing who they are. Needs multi-speaker audio with timed speaker labels, which is a different annotation entirely — and critically, needs overlapping speech if you want it to work on real conversations.

A side-by-side comparison of two speaker diarisation architectures. The cascaded system is drawn as a chain of separate blocks - speech activity detection, segmentation, speaker embedding extraction, clustering and target-speaker voice activity detection. The end-to-end system is a single block taking the audio directly to per-speaker activity outputs, with overlapping speaker regions shown as simultaneously active output tracks. — Cascaded against end-to-end diarisation. In the cascade, clustering gives each segment one speaker and a final target-speaker voice activity detection stage has to recover the overlap; the end-to-end model outputs per-speaker tracks that can be active at once. Image: NVIDIA/NeMo · Apache-2.0 repository

The property that matters: sessions per speaker

This is the thing to check first and the reason most corpora disappoint.

A speaker model must learn what varies within a person versus between people. Within-speaker variation is large: your voice differs with a cold, time of day, mood, microphone, room, and how close you are to it.

If each speaker appears in exactly one recording session, the model cannot separate speaker identity from session characteristics. It learns to recognise the recording — the room, the microphone, the channel — and reports excellent accuracy on a test set drawn from the same sessions. In deployment, where someone calls from a different phone, it fails.

So the question to ask of a candidate dataset is not how many speakers but how many separate sessions per speaker, and whether those sessions differ in channel and condition. Multiple sessions across varied conditions is the property; raw speaker count is a distraction.

Diarisation fails on overlap

If you are building diarisation, this is the single most important thing to check in your data.

Four stacked timelines from 100 to 122 seconds labelled reference, diarization, resegmentation and heuristic, each with blue and red bars for two speakers. The reference and resegmentation rows have two lanes where speakers overlap; the diarization row has a single lane. Three regions are shaded grey. — Twenty seconds of a DIHARD III recording: the reference annotation (top, two speakers in blue and red), a standard clustering diarisation (second row), the authors' overlap-aware resegmentation (third) and a simple heuristic (bottom). Clustering gives each moment one speaker, so overlapping speech is lost; recovering it takes a model trained on data that marks overlap. Image: Figure 3 of the paper · CC BY 4.0 · from Bredin & Laurent 2021, “End-to-end speaker segmentation for overlap-aware resegmentation”, arXiv:2104.04045

Real conversations contain simultaneous speech: interruptions, back-channel agreement, crosstalk in meetings. Depending on the setting, overlapping speech can be a substantial fraction of a recording, and it is where diarisation systems do most of their failing.

Many datasets either avoid overlap by construction — read speech, turn-taking prompts — or do not annotate it, labelling only the dominant speaker. A system trained and evaluated on such data will report a good error rate and then struggle on a real meeting.

So check whether overlapping regions are annotated as overlapping, and whether your evaluation metric accounts for them. If your deployment target is meetings or calls, deliberately include and measure overlap, because the aggregate error rate will otherwise hide the only hard part of the problem.

Two stacked speaker timelines for 600 to 630 seconds of AMI meeting ES2004a. The reference shows four speakers on three rows, with short overlapping turns on the second and third rows near 604, 606, 609 and 622 seconds and a red MEO015 turn from about 617.5 to 619.5 seconds. The pipeline output below has only two rows: the short overlapping turns are missing, MEO015's turn is gone, and a few spurious grey fragments labelled SPEAKER_03 appear near 622 to 624 seconds. — Thirty seconds of a four-person meeting, annotated by hand and then diarised automatically. The long turns survive; what disappears is the overlap: the back-channels on the lower rows and a two-second interjection from a fourth speaker. The whole-file error rate of 19.8% gives no hint that this is where it went. Images: pyannote/pyannote-audio · MIT repository; annotations from the AMI Meeting Corpus (CC BY 4.0)

Biometric obligations apply

Voice is personal data and in several regimes biometric data, which brings heightened obligations — the same position as face data.

A dataset's permissive licence does not resolve this, because the licence is granted by the compiler while the rights belong to the speakers. And consent given for speech recognition research does not straightforwardly extend to training a biometric identification system, which is a materially different use that many older corpora's consent documentation never contemplated.

If you are building voice authentication specifically, treat the data question as a compliance question from the start rather than a search problem. For an internal system, recording your own enrolment data with explicit consent is both cleaner legally and better matched to your actual channel conditions — which, given the session-variability point above, is also the technically better answer.

Questions people ask

How many speakers do I need?

For verification, more speakers helps the model learn general speaker-discriminative features — hundreds upward is typical. But sessions per speaker is the binding constraint: a hundred speakers with five varied sessions each beats a thousand with one.

Can I use an ASR corpus with speaker labels?

Only if it has multiple sessions per speaker under varied conditions. Most large recognition corpora do not — each contributor records once, so speaker identity and session characteristics are confounded. Check before committing.

What error rate should I expect for diarisation?

Highly dependent on conditions, and overlap is the dominant factor. Published figures on clean two-speaker audio are much better than on multi-party meeting recordings. Compare numbers only across the same conditions, and check whether overlap was scored.

Do I need separation as well as diarisation?

If overlapping speech matters and you need the words, yes — diarisation tells you who spoke when, separation recovers each voice from the mixture. They are separate stages, and transcribing overlapped speech generally needs the second.

Keep reading

Audio & speechNine open audio datasets for speech and sound models Audio & speechSpeech recognition datasets Audio & speechAudio event and sound classification datasets