Skip to main content

DATASET 01 · Phonetics

Mandarin Multimodal Read-Speech and Conversation Corpus

Read-speech and natural conversation from 2,800 native speakers, delivered with synchronized audio, video, transcripts and prosodic annotation.

SpeechVideoTextPhoneticsLanguage teaching
Mandarin Multimodal Read-Speech and Conversation Corpus · cover artwork
Scale
1,240 hours
Participants
2,800 native speakers
Languages
Mandarin (8 dialect-area accents)
Annotation
Phoneme-level forced alignment, Prosodic boundaries and prominence, Verbatim transcription, Emotion and stance labels
Formats
WAV / TextGrid / ELAN / JSON
Sync
Audio-video sync error ≤ 2 ms
License
Research license / Commercial license
Updated
September 12, 2026

01 / Composition

Scale, key metrics and data streams

Foundational Mandarin data, from controlled reading to natural dialogue

1,240 h
Total duration
2,800
Speakers
8
Dialect-area accents
Data composition and specifications
StreamSpecificationNotes
Audio48 kHz / 24-bitDual-channel with a low noise floor; the primary channel is a headset microphone
Video1920 × 1080 · 30 fpsFront and side cameras, synchronized with a clapperboard
TextUTF-8 / JSONVerbatim transcript with phoneme-boundary alignment

The table lists capture parameters for a sample batch; final specifications follow the documentation delivered with the data.

02 / Dataset notes

Collection, annotation and quality control

The notes below explain how the data was collected, annotated and verified, and can be used as a basis for methodological citation and proposal assessment.

Overview

The Mandarin Multimodal Read-Speech and Conversation Corpus covers 2,800 native speakers recorded between April 2023 and March 2026, totaling 1,240 hours of valid speech. The data falls into three parts: 420 hours of controlled reading, 380 hours of semi-structured interviews and 440 hours of natural conversation. Speakers are sampled by gender, four age bands (18–25, 26–40, 41–60 and over 60) and eight dialect-area accents. Each speaker contributes at least 25 minutes of valid speech, balancing individual acoustic characteristics against group representativeness.

Conversations are recorded in quiet laboratories and semi-anechoic rooms. Two-person dialogues are guided either by free topics or by assigned tasks, so batches remain comparable while speech stays natural.

Collection protocol

Audio is captured at 48 kHz / 24-bit through two channels. The primary channel uses a headset microphone, while a reference channel records the room noise floor. Two cameras record at 1920 × 1080 · 30 fps: the front camera captures lip movement and facial expression, and the side camera captures posture. All device clocks are calibrated before collection; every device is timed by the same time-base server, and the clapperboard frame is written into the metadata.

Participants are recruited through universities and community channels. Each signs an informed consent form and completes a speaker background questionnaire covering birthplace, upbringing, education and languages used. Anything potentially identifying is redacted and logged on the day of recording. A single session is capped at 90 minutes with a 10-minute break in the middle, limiting how vocal fatigue affects acoustic parameters.

Annotation

Annotation has four layers and is carried out by trained annotators. Each layer has its own manual and example library.

LayerContentScaleReview
PhonemeInitial and final boundaries with confidence62 phoneme–tone combinationsAutomatic alignment + 15% manual spot-check
ProsodicProsodic boundaries, prominence, intonation phrases4 boundary levels, 3 prominence levelsDual annotation, arbitration review
TranscriptionVerbatim transcription, disfluency markers, speech slips18 disfluency label types20% double-blind review
EmotionEmotion category and stance intensity7 emotions, 4 intensity levelsDual annotation, κ reported

Quality & verification

The median audio-video sync error is 1.4 ms, with the 95th percentile at or below 2 ms. Audio signal-to-noise ratio is at least 42 dB, verbatim transcription character accuracy is 99.2%, and prosodic boundary agreement reaches κ = 0.89. Twelve percent of every batch is double-blind reviewed; a batch that fails is re-annotated in full and never delivered. Deliverables include a sync measurement report, noise statistics, an annotation agreement report and a metadata dictionary, so every metric can be traced back to its batch.

Research questions

  • How dialect-area accents affect the perception and production of Mandarin tonal categories.
  • How prosodic boundaries correspond to syntactic boundaries in natural conversation.
  • How strongly emotion labels associate with acoustic parameters such as F0, speech rate and energy.
  • Where ASR systems differ in robustness under cross-dialect accent conditions.
  • How read speech and spontaneous speech differ systematically in pause patterns.

Access & licensing

The research license serves universities and non-profit research institutions; the commercial license serves speech technology, foundation-model and language-teaching product teams. Delivery includes the raw audio and video, aligned annotated text, annotation manuals and data documentation. You can request a 2-hour sample batch through the sample data page, or describe your research question and use case on the contact page to receive a quote and delivery timeline for a custom subset.

03 / Typical use cases

Questions this dataset can answer

The directions below are typical research and product applications; a specific analysis plan depends on the study design and statistical method.

  • 01

    ASR robustness evaluation

  • 02

    Prosody and intonation research

  • 03

    Language teaching and pronunciation assessment

  • 04

    Emotion and stance research

NEXT STEP

Need a different collection setting or annotation scheme?

An existing dataset can be licensed as it stands, or extended with additional speakers, scenarios and annotation layers around your study design.

Reply within one business day · NDA and data licensing supported · Dedicated academic track