DATASET 01 · Phonetics
Mandarin Multimodal Read-Speech and Conversation Corpus
Read-speech and natural conversation from 2,800 native speakers, delivered with synchronized audio, video, transcripts and prosodic annotation.
- Scale
- 1,240 hours
- Participants
- 2,800 native speakers
- Languages
- Mandarin (8 dialect-area accents)
- Annotation
- Phoneme-level forced alignment, Prosodic boundaries and prominence, Verbatim transcription, Emotion and stance labels
- Formats
- WAV / TextGrid / ELAN / JSON
- Sync
- Audio-video sync error ≤ 2 ms
- License
- Research license / Commercial license
- Updated
- September 12, 2026
01 / Composition
Scale, key metrics and data streams
Foundational Mandarin data, from controlled reading to natural dialogue
- 1,240 h
- Total duration
- 2,800
- Speakers
- 8
- Dialect-area accents
| Stream | Specification | Notes |
|---|---|---|
| Audio | 48 kHz / 24-bit | Dual-channel with a low noise floor; the primary channel is a headset microphone |
| Video | 1920 × 1080 · 30 fps | Front and side cameras, synchronized with a clapperboard |
| Text | UTF-8 / JSON | Verbatim transcript with phoneme-boundary alignment |
The table lists capture parameters for a sample batch; final specifications follow the documentation delivered with the data.
02 / Dataset notes
Collection, annotation and quality control
The notes below explain how the data was collected, annotated and verified, and can be used as a basis for methodological citation and proposal assessment.
Overview
The Mandarin Multimodal Read-Speech and Conversation Corpus covers 2,800 native speakers recorded between April 2023 and March 2026, totaling 1,240 hours of valid speech. The data falls into three parts: 420 hours of controlled reading, 380 hours of semi-structured interviews and 440 hours of natural conversation. Speakers are sampled by gender, four age bands (18–25, 26–40, 41–60 and over 60) and eight dialect-area accents. Each speaker contributes at least 25 minutes of valid speech, balancing individual acoustic characteristics against group representativeness.
Conversations are recorded in quiet laboratories and semi-anechoic rooms. Two-person dialogues are guided either by free topics or by assigned tasks, so batches remain comparable while speech stays natural.
Collection protocol
Audio is captured at 48 kHz / 24-bit through two channels. The primary channel uses a headset microphone, while a reference channel records the room noise floor. Two cameras record at 1920 × 1080 · 30 fps: the front camera captures lip movement and facial expression, and the side camera captures posture. All device clocks are calibrated before collection; every device is timed by the same time-base server, and the clapperboard frame is written into the metadata.
Participants are recruited through universities and community channels. Each signs an informed consent form and completes a speaker background questionnaire covering birthplace, upbringing, education and languages used. Anything potentially identifying is redacted and logged on the day of recording. A single session is capped at 90 minutes with a 10-minute break in the middle, limiting how vocal fatigue affects acoustic parameters.
Annotation
Annotation has four layers and is carried out by trained annotators. Each layer has its own manual and example library.
| Layer | Content | Scale | Review |
|---|---|---|---|
| Phoneme | Initial and final boundaries with confidence | 62 phoneme–tone combinations | Automatic alignment + 15% manual spot-check |
| Prosodic | Prosodic boundaries, prominence, intonation phrases | 4 boundary levels, 3 prominence levels | Dual annotation, arbitration review |
| Transcription | Verbatim transcription, disfluency markers, speech slips | 18 disfluency label types | 20% double-blind review |
| Emotion | Emotion category and stance intensity | 7 emotions, 4 intensity levels | Dual annotation, κ reported |
Quality & verification
The median audio-video sync error is 1.4 ms, with the 95th percentile at or below 2 ms. Audio signal-to-noise ratio is at least 42 dB, verbatim transcription character accuracy is 99.2%, and prosodic boundary agreement reaches κ = 0.89. Twelve percent of every batch is double-blind reviewed; a batch that fails is re-annotated in full and never delivered. Deliverables include a sync measurement report, noise statistics, an annotation agreement report and a metadata dictionary, so every metric can be traced back to its batch.
Research questions
- How dialect-area accents affect the perception and production of Mandarin tonal categories.
- How prosodic boundaries correspond to syntactic boundaries in natural conversation.
- How strongly emotion labels associate with acoustic parameters such as F0, speech rate and energy.
- Where ASR systems differ in robustness under cross-dialect accent conditions.
- How read speech and spontaneous speech differ systematically in pause patterns.
Access & licensing
The research license serves universities and non-profit research institutions; the commercial license serves speech technology, foundation-model and language-teaching product teams. Delivery includes the raw audio and video, aligned annotated text, annotation manuals and data documentation. You can request a 2-hour sample batch through the sample data page, or describe your research question and use case on the contact page to receive a quote and delivery timeline for a custom subset.
03 / Typical use cases
Questions this dataset can answer
The directions below are typical research and product applications; a specific analysis plan depends on the study design and statistical method.
- 01
ASR robustness evaluation
- 02
Prosody and intonation research
- 03
Language teaching and pronunciation assessment
- 04
Emotion and stance research
04 / Related datasets
Datasets that combine well
A single research question often calls for a cross-modal combination; the entries below share scenarios, speakers or annotation schemes with this dataset.
NEXT STEP
Need a different collection setting or annotation scheme?
An existing dataset can be licensed as it stands, or extended with additional speakers, scenarios and annotation layers around your study design.
Reply within one business day · NDA and data licensing supported · Dedicated academic track