DATASET 06 · Affective computing
Multimodal Emotion and Paralanguage Corpus
Speech, facial, physiological and self-report data from 1,200 participants, covering 7 emotion categories and multiple paralinguistic styles.
- Scale
- 380 hours
- Participants
- 1,200 participants
- Languages
- Mandarin (5 emotional contexts)
- Annotation
- Emotion category and intensity, Paralinguistic features, Facial action units, Physiological arousal
- Formats
- WAV / MP4 / CSV / JSON
- Sync
- Audio-video and physiological signal sync error ≤ 2 ms
- License
- Research license / Commercial license (ethics review required)
- Updated
- September 2, 2026
01 / Composition
Scale, key metrics and data streams
Restoring emotion from a single modality to multi-channel evidence
- 380 h
- Total duration
- 1,200
- Participants
- 7
- Emotion categories
- 4
- Physiological channels
| Stream | Specification | Notes |
|---|---|---|
| Audio | 48 kHz / 24-bit | Headset microphone, preserving breathing and pauses |
| Video | 1920 × 1080 · 60 fps | Front camera covering the face and upper body |
| Physiology | 1,000 Hz | Four channels: electrodermal activity, heart rate, respiration and facial EMG |
| Text | UTF-8 / JSON | Elicitation materials, self-report scales and transcripts |
The table lists capture parameters for a sample batch; final specifications follow the documentation delivered with the data.
02 / Dataset notes
Collection, annotation and quality control
The notes below explain how the data was collected, annotated and verified, and can be used as a basis for methodological citation and proposal assessment.
Overview
The Multimodal Emotion and Paralanguage Corpus covers 1,200 participants and 380 hours recorded between June 2024 and June 2026. Data is gathered through both situational elicitation and natural expression, covering seven emotions — happiness, sadness, anger, fear, surprise, disgust and neutrality — with each emotion represented in positive, neutral and negative contexts.
Participants are stratified by gender, age and occupational background. Of these, 240 completed two collection rounds, which supports assessment of the temporal stability of individual emotional expression. Paralinguistic style variables include speech rate, loudness, pitch range and pause density.
Collection protocol
Collection takes place in an acoustically treated laboratory. Audio is recorded at 48 kHz / 24-bit, video at 1920 × 1080 · 60 fps from a front camera, and four physiological channels — electrodermal activity, heart rate, respiration and facial EMG — at 1,000 Hz. All channels feed the same acquisition host and are time-stamped by a trigger box to guarantee cross-channel alignment.
Emotion induction uses pre-tested standardized materials, including short videos, images and role-play tasks. After each trial, participants rate valence and arousal on nine-point scales. The protocol follows ethics requirements throughout, participants may withdraw at any time, and narratives involving personal experience are content-reviewed and de-identified before ingest.
Annotation
Annotation combines three layers of evidence: subjective self-report, expert coding and automatic features.
| Layer | Content | Scale | Review |
|---|---|---|---|
| Self-report | Valence, arousal, dominance | All trials | Scale quality control |
| Expert | Emotion category, intensity, mixed emotions | About 42,000 clips | Dual annotation, arbitration review |
| Facial | Action units (AU) and intensity | All videos | Automatic detection + 15% spot-check |
| Paralinguistic | Speech rate, pitch, energy, pauses | All audio | Algorithmic extraction + spot-check |
Quality & verification
The median sync error across audio, video and physiological signals is 1.1 ms, with the 95th percentile at or below 2 ms. Expert emotion annotation agreement is κ = 0.83, facial action unit detection agrees with manual coding 91.7% of the time, and the physiological artifact rate is below 4%. Twelve percent of clips in every batch are double-blind reviewed, and trials where emotion induction failed — self-report inconsistent with the expected category — are flagged separately. Deliverables include the elicitation material list, scale coding, feature extraction scripts and a sync report.
Research questions
- The performance gap between single-modality and multimodal emotion recognition.
- The non-linear relationship between emotion intensity and pitch range or speech rate.
- How well physiological arousal predicts vocal emotion expression.
- Generalization boundaries of cross-subject and cross-gender emotion recognition models.
- Inconsistent expression of mixed emotions across channels.
Access & licensing
The research license serves affective computing, linguistics and psychology teams; the commercial license serves human-computer interaction, intelligent cockpit and mental health products. Data is delivered as WAV, MP4, CSV and JSON, with feature tables and raw signals provided separately. Because physiological and facial data are involved, applicants must provide ethics approval and a data security plan. You can inspect clip-level fields through the sample data page, or discuss licensing scope on the contact page.
03 / Typical use cases
Questions this dataset can answer
The directions below are typical research and product applications; a specific analysis plan depends on the study design and statistical method.
- 01
Multimodal emotion recognition benchmark
- 02
Emotion and acoustic feature association
- 03
Cross-subject emotion generalization
- 04
Empathic human-computer interaction design
04 / Related datasets
Datasets that combine well
A single research question often calls for a cross-modal combination; the entries below share scenarios, speakers or annotation schemes with this dataset.
NEXT STEP
Need a different collection setting or annotation scheme?
An existing dataset can be licensed as it stands, or extended with additional speakers, scenarios and annotation layers around your study design.
Reply within one business day · NDA and data licensing supported · Dedicated academic track