Skip to main content

DATASET 06 · Affective computing

Multimodal Emotion and Paralanguage Corpus

Speech, facial, physiological and self-report data from 1,200 participants, covering 7 emotion categories and multiple paralinguistic styles.

SpeechVideoPhysiologyMultimodalAffective computingParalinguistics
Multimodal Emotion and Paralanguage Corpus · cover artwork
Scale
380 hours
Participants
1,200 participants
Languages
Mandarin (5 emotional contexts)
Annotation
Emotion category and intensity, Paralinguistic features, Facial action units, Physiological arousal
Formats
WAV / MP4 / CSV / JSON
Sync
Audio-video and physiological signal sync error ≤ 2 ms
License
Research license / Commercial license (ethics review required)
Updated
September 2, 2026

01 / Composition

Scale, key metrics and data streams

Restoring emotion from a single modality to multi-channel evidence

380 h
Total duration
1,200
Participants
7
Emotion categories
4
Physiological channels
Data composition and specifications
StreamSpecificationNotes
Audio48 kHz / 24-bitHeadset microphone, preserving breathing and pauses
Video1920 × 1080 · 60 fpsFront camera covering the face and upper body
Physiology1,000 HzFour channels: electrodermal activity, heart rate, respiration and facial EMG
TextUTF-8 / JSONElicitation materials, self-report scales and transcripts

The table lists capture parameters for a sample batch; final specifications follow the documentation delivered with the data.

02 / Dataset notes

Collection, annotation and quality control

The notes below explain how the data was collected, annotated and verified, and can be used as a basis for methodological citation and proposal assessment.

Overview

The Multimodal Emotion and Paralanguage Corpus covers 1,200 participants and 380 hours recorded between June 2024 and June 2026. Data is gathered through both situational elicitation and natural expression, covering seven emotions — happiness, sadness, anger, fear, surprise, disgust and neutrality — with each emotion represented in positive, neutral and negative contexts.

Participants are stratified by gender, age and occupational background. Of these, 240 completed two collection rounds, which supports assessment of the temporal stability of individual emotional expression. Paralinguistic style variables include speech rate, loudness, pitch range and pause density.

Collection protocol

Collection takes place in an acoustically treated laboratory. Audio is recorded at 48 kHz / 24-bit, video at 1920 × 1080 · 60 fps from a front camera, and four physiological channels — electrodermal activity, heart rate, respiration and facial EMG — at 1,000 Hz. All channels feed the same acquisition host and are time-stamped by a trigger box to guarantee cross-channel alignment.

Emotion induction uses pre-tested standardized materials, including short videos, images and role-play tasks. After each trial, participants rate valence and arousal on nine-point scales. The protocol follows ethics requirements throughout, participants may withdraw at any time, and narratives involving personal experience are content-reviewed and de-identified before ingest.

Annotation

Annotation combines three layers of evidence: subjective self-report, expert coding and automatic features.

LayerContentScaleReview
Self-reportValence, arousal, dominanceAll trialsScale quality control
ExpertEmotion category, intensity, mixed emotionsAbout 42,000 clipsDual annotation, arbitration review
FacialAction units (AU) and intensityAll videosAutomatic detection + 15% spot-check
ParalinguisticSpeech rate, pitch, energy, pausesAll audioAlgorithmic extraction + spot-check

Quality & verification

The median sync error across audio, video and physiological signals is 1.1 ms, with the 95th percentile at or below 2 ms. Expert emotion annotation agreement is κ = 0.83, facial action unit detection agrees with manual coding 91.7% of the time, and the physiological artifact rate is below 4%. Twelve percent of clips in every batch are double-blind reviewed, and trials where emotion induction failed — self-report inconsistent with the expected category — are flagged separately. Deliverables include the elicitation material list, scale coding, feature extraction scripts and a sync report.

Research questions

  • The performance gap between single-modality and multimodal emotion recognition.
  • The non-linear relationship between emotion intensity and pitch range or speech rate.
  • How well physiological arousal predicts vocal emotion expression.
  • Generalization boundaries of cross-subject and cross-gender emotion recognition models.
  • Inconsistent expression of mixed emotions across channels.

Access & licensing

The research license serves affective computing, linguistics and psychology teams; the commercial license serves human-computer interaction, intelligent cockpit and mental health products. Data is delivered as WAV, MP4, CSV and JSON, with feature tables and raw signals provided separately. Because physiological and facial data are involved, applicants must provide ethics approval and a data security plan. You can inspect clip-level fields through the sample data page, or discuss licensing scope on the contact page.

03 / Typical use cases

Questions this dataset can answer

The directions below are typical research and product applications; a specific analysis plan depends on the study design and statistical method.

  • 01

    Multimodal emotion recognition benchmark

  • 02

    Emotion and acoustic feature association

  • 03

    Cross-subject emotion generalization

  • 04

    Empathic human-computer interaction design

NEXT STEP

Need a different collection setting or annotation scheme?

An existing dataset can be licensed as it stands, or extended with additional speakers, scenarios and annotation layers around your study design.

Reply within one business day · NDA and data licensing supported · Dedicated academic track