Speech emotion datasets: what the labels actually measure

Intended, perceived and felt are three different labels

Every emotion corpus records one of three things, usually without saying which. The intended emotion is what an actor was asked to portray. The perceived emotion is what listeners heard. The felt emotion is what the speaker experienced, which no public corpus measures directly.

The gap between the first two is large. CREMA-D had every clip rated by crowd workers — 2,443 raters, close to ten ratings per clip — and in the audio-only condition the raters chose the emotion the actor had been asked to express 40.9% of the time. With video as well it rose to 63.6%. If listeners hearing a deliberate performance agree with the script four times in ten, the label is not ground truth in the sense a bounding box is.

So check which label a dataset ships. Intended labels are clean and balanced and describe the script. Perceived labels are noisy and describe the audience. For a product that reacts to how a customer sounds, perceived is usually what you want — and it is worth keeping the full distribution of rater votes as a soft label rather than collapsing it to a single class.

Acted or natural

Most widely used emotion corpora are acted. Professional actors read fixed sentences in each target emotion, which gives tidy, balanced classes and lexically identical utterances across emotions — useful, because a model cannot cheat by reading the words.

The cost is that acted emotion is exaggerated and evenly distributed. Real speech is mostly neutral, the emotional moments are subtle, and the class balance is nothing like a studio corpus. A model trained on acted data and deployed on calls tends to see strong emotion where there is little.

Naturalistic corpora fix this by sampling real speech and labelling it afterwards. The trade is a heavily imbalanced label distribution, more disagreement between raters, and — because the words now vary — the risk that a model learns from what was said rather than how. If your audio model's accuracy collapses when you replace the speech with a neutral transcript read aloud, it was reading the words.

Categories or dimensions

Categorical labels pick from a small set — anger, happiness, sadness, neutral and a few more. They are easy to explain and hard to apply: many real utterances are a mixture, and the sets differ between corpora, so a merged dataset needs a mapping that inevitably loses information.

Dimensional labels rate each utterance on continuous scales, most often arousal (how activated), valence (how positive) and dominance (how in control). They describe mixtures naturally and transfer between corpora better, at the price of being harder to turn into a product decision.

The better corpora carry both. If you are combining datasets, dimensions are the safer common ground; if you need categories at the end, derive them from your own thresholds on arousal and valence rather than inheriting four different taxonomies.

The corpora people use, and the terms that come with them

The four below are the ones most papers report on. They are named here so you can assess them, and the access terms are as each project publishes them — read the source before you build on any of them.

RAVDESS — 7,356 files from 24 actors, twelve female and twelve male, in speech and song, with calm, happy, sad, angry, fearful, surprise, disgust and neutral. Acted, with audio and video of the same performance. Released under CC BY-NC-SA 4.0, so non-commercial and share-alike; the authors sell commercial licences separately.

CREMA-D — 7,442 clips from 91 actors of varied ethnic background, twelve sentences in six states: anger, disgust, fear, happy, neutral and sad. Its distinguishing feature is the perceived labels described above, collected separately for audio, video and both. Published under the Open Database License, with contents under the Database Contents License.

IEMOCAP — about twelve hours of two-person interaction between ten actors, partly scripted and partly improvised, with both categorical and dimensional labels. Access is by a release form from USC, and the licence is for internal research only, with no commercial use.

MSP-Podcast — naturalistic speech taken from podcasts, around 409 hours and 267,905 speaking turns from 3,641 speakers in its 2.0 release. Each turn has a primary emotion, secondary emotions, and arousal, valence and dominance on a seven-point scale, from at least five raters. It is distributed through data transfer agreements with research groups.

Split by speaker, or the score is meaningless

Acted corpora have few speakers — 24, 91 and 10 in RAVDESS, CREMA-D and IEMOCAP — and a lot of clips from each. Split them at random and the same voice lands in training and test, so the model can score well by recognising the speaker's way of performing an emotion rather than the emotion.

Hold speakers out entirely. With a small corpus that means a leave-one-speaker-out or k-fold-by-speaker scheme rather than a single split, and the variance between folds is worth reporting alongside the mean, because with ten or twenty-four people one unusual performer moves the number a long way.

In Dataset Finder

Of the corpora above, RAVDESS is in Dataset Finder's verified set. Our speech and audio datasets post covers the wider audio catalogue, and the speaker identification post covers the session problem, which applies here with equal force.

Questions people ask