DATASET 03 · Medical linguistics
Clinical Conversation Multimodal Corpus
Audio, video and de-identified transcripts from 1,450 real consultations across 6 departments, supporting clinician–patient communication and clinical language research.
- Scale
- 620 hours
- Participants
- 1,450 consultations
- Languages
- Mandarin (3 dialect-area accents)
- Annotation
- Turn-taking and topic shifts, Empathy and reassurance markers, Medical entity annotation, Nonverbal behaviour coding
- Formats
- WAV / MP4 / TextGrid / JSON
- Sync
- Audio-video sync error ≤ 2 ms
- License
- Research license / Commercial license (ethics review required)
- Updated
- July 18, 2026
01 / Composition
Scale, key metrics and data streams
Recording language, facial expression and interaction in real consultations
- 620 h
- Total duration
- 1,450
- Consultations
- 6
- Departments
| Stream | Specification | Notes |
|---|---|---|
| Audio | 48 kHz / 24-bit · dual channel | Independent clinician and patient channels, preserving natural overlap |
| Video | 1920 × 1080 · 30 fps | Two camera angles covering both upper bodies |
| Text | UTF-8 / JSON | De-identified verbatim transcript with speaker diarization |
The table lists capture parameters for a sample batch; final specifications follow the documentation delivered with the data.
02 / Dataset notes
Collection, annotation and quality control
The notes below explain how the data was collected, annotated and verified, and can be used as a basis for methodological citation and proposal assessment.
Overview
The Clinical Conversation Multimodal Corpus contains 1,450 real outpatient consultations totaling 620 hours. It covers six departments — internal medicine, surgery, paediatrics, dermatology, psychiatry and rehabilitation — recorded between January 2024 and April 2026. A consultation lasts 25.7 minutes on average, ranging from 6 to 58 minutes, and each recording preserves the complete visit from the moment the patient sits down to the moment they leave.
The corpus is stratified by department, consultation type (first visit, follow-up, return visit) and clinician seniority. A single clinician contributes between 10 and 30 consultations, which supports comparison of individual communication style against stable patterns.
Collection protocol
Each consultation room is equipped with dual-channel lavalier microphones. Clinician and patient are recorded on independent channels at 48 kHz / 24-bit. Two 1920 × 1080 · 30 fps cameras record the upper bodies and hand movements of both parties. The hospital ethics committee approves the protocol before collection, and patients and clinicians sign separate consent forms; either party can withdraw unconditionally at any point.
To limit privacy risk, screens showing medical records, test reports and identity information are masked at the point of capture. Transcription follows a strict de-identification procedure: names, places, institutions, dates and identifiers are replaced with typed placeholders, and the replacement mapping is never delivered with the data.
Annotation
Annotation is carried out by annotators with a medical background, and the medical entity layer is reviewed by clinicians.
| Layer | Content | Scale | Review |
|---|---|---|---|
| Conversation | Turn segmentation, topic shifts, interruptions and overlaps | About 380,000 turns | 20% double-blind review |
| Communication | Empathy, reassurance, explanation, shared decision-making | 12 behaviour classes | Dual annotation, κ reported |
| Entities | Symptoms, examinations, diagnoses, medication, dosage | About 210,000 entities | 15% clinician spot-check |
| Nonverbal | Nodding, frowning, gesture, gaze | 8 behaviour classes | Dual annotation of key segments |
Quality & verification
The median audio-video sync error is 1.2 ms, with the 95th percentile at or below 2 ms. Transcription character accuracy is 98.9%, communication behaviour agreement is κ = 0.88, and medical entity F1 is 0.94. Ten percent of consultations in every batch are double-blind reviewed, and an independent team rescans de-identification output, with a miss rate below 0.1%. Deliverables include an ethics approval summary, a de-identification statement and an annotation agreement report.
Research questions
- How clinician empathy expression relates to subsequent patient adherence.
- How consultation structure and information-transfer efficiency differ across departments.
- When nonverbal signals of hesitation and avoidance appear in patients.
- How factually consistent clinical dialogue summarization models are on real consultations.
- How dialect accents affect mutual understanding and repeated explanation.
Access & licensing
This dataset is available only to research projects that pass ethics review and to compliant commercial settings. The research license requires a study protocol, ethics approval and a data management plan; the commercial license additionally requires a data processing agreement and prohibits re-identification and redistribution of source video. Data is delivered as WAV, MP4, TextGrid and JSON, filtered by department and consultation type. You can inspect the de-identified field structure through the sample data page, or arrange an ethics and licensing conversation on the contact page.
03 / Typical use cases
Questions this dataset can answer
The directions below are typical research and product applications; a specific analysis plan depends on the study design and statistical method.
- 01
Clinician–patient communication quality assessment
- 02
Empathy language and patient satisfaction research
- 03
Clinical dialogue summarization and entity extraction
- 04
Consultation workflow and communication training
04 / Related datasets
Datasets that combine well
A single research question often calls for a cross-modal combination; the entries below share scenarios, speakers or annotation schemes with this dataset.
NEXT STEP
Need a different collection setting or annotation scheme?
An existing dataset can be licensed as it stands, or extended with additional speakers, scenarios and annotation layers around your study design.
Reply within one business day · NDA and data licensing supported · Dedicated academic track