Skip to main content

DATASET 03 · Medical linguistics

Clinical Conversation Multimodal Corpus

Audio, video and de-identified transcripts from 1,450 real consultations across 6 departments, supporting clinician–patient communication and clinical language research.

SpeechVideoTextMedical linguisticsConversation analysis
Clinical Conversation Multimodal Corpus · cover artwork
Scale
620 hours
Participants
1,450 consultations
Languages
Mandarin (3 dialect-area accents)
Annotation
Turn-taking and topic shifts, Empathy and reassurance markers, Medical entity annotation, Nonverbal behaviour coding
Formats
WAV / MP4 / TextGrid / JSON
Sync
Audio-video sync error ≤ 2 ms
License
Research license / Commercial license (ethics review required)
Updated
July 18, 2026

01 / Composition

Scale, key metrics and data streams

Recording language, facial expression and interaction in real consultations

620 h
Total duration
1,450
Consultations
6
Departments
Data composition and specifications
StreamSpecificationNotes
Audio48 kHz / 24-bit · dual channelIndependent clinician and patient channels, preserving natural overlap
Video1920 × 1080 · 30 fpsTwo camera angles covering both upper bodies
TextUTF-8 / JSONDe-identified verbatim transcript with speaker diarization

The table lists capture parameters for a sample batch; final specifications follow the documentation delivered with the data.

02 / Dataset notes

Collection, annotation and quality control

The notes below explain how the data was collected, annotated and verified, and can be used as a basis for methodological citation and proposal assessment.

Overview

The Clinical Conversation Multimodal Corpus contains 1,450 real outpatient consultations totaling 620 hours. It covers six departments — internal medicine, surgery, paediatrics, dermatology, psychiatry and rehabilitation — recorded between January 2024 and April 2026. A consultation lasts 25.7 minutes on average, ranging from 6 to 58 minutes, and each recording preserves the complete visit from the moment the patient sits down to the moment they leave.

The corpus is stratified by department, consultation type (first visit, follow-up, return visit) and clinician seniority. A single clinician contributes between 10 and 30 consultations, which supports comparison of individual communication style against stable patterns.

Collection protocol

Each consultation room is equipped with dual-channel lavalier microphones. Clinician and patient are recorded on independent channels at 48 kHz / 24-bit. Two 1920 × 1080 · 30 fps cameras record the upper bodies and hand movements of both parties. The hospital ethics committee approves the protocol before collection, and patients and clinicians sign separate consent forms; either party can withdraw unconditionally at any point.

To limit privacy risk, screens showing medical records, test reports and identity information are masked at the point of capture. Transcription follows a strict de-identification procedure: names, places, institutions, dates and identifiers are replaced with typed placeholders, and the replacement mapping is never delivered with the data.

Annotation

Annotation is carried out by annotators with a medical background, and the medical entity layer is reviewed by clinicians.

LayerContentScaleReview
ConversationTurn segmentation, topic shifts, interruptions and overlapsAbout 380,000 turns20% double-blind review
CommunicationEmpathy, reassurance, explanation, shared decision-making12 behaviour classesDual annotation, κ reported
EntitiesSymptoms, examinations, diagnoses, medication, dosageAbout 210,000 entities15% clinician spot-check
NonverbalNodding, frowning, gesture, gaze8 behaviour classesDual annotation of key segments

Quality & verification

The median audio-video sync error is 1.2 ms, with the 95th percentile at or below 2 ms. Transcription character accuracy is 98.9%, communication behaviour agreement is κ = 0.88, and medical entity F1 is 0.94. Ten percent of consultations in every batch are double-blind reviewed, and an independent team rescans de-identification output, with a miss rate below 0.1%. Deliverables include an ethics approval summary, a de-identification statement and an annotation agreement report.

Research questions

  • How clinician empathy expression relates to subsequent patient adherence.
  • How consultation structure and information-transfer efficiency differ across departments.
  • When nonverbal signals of hesitation and avoidance appear in patients.
  • How factually consistent clinical dialogue summarization models are on real consultations.
  • How dialect accents affect mutual understanding and repeated explanation.

Access & licensing

This dataset is available only to research projects that pass ethics review and to compliant commercial settings. The research license requires a study protocol, ethics approval and a data management plan; the commercial license additionally requires a data processing agreement and prohibits re-identification and redistribution of source video. Data is delivered as WAV, MP4, TextGrid and JSON, filtered by department and consultation type. You can inspect the de-identified field structure through the sample data page, or arrange an ethics and licensing conversation on the contact page.

03 / Typical use cases

Questions this dataset can answer

The directions below are typical research and product applications; a specific analysis plan depends on the study design and statistical method.

  • 01

    Clinician–patient communication quality assessment

  • 02

    Empathy language and patient satisfaction research

  • 03

    Clinical dialogue summarization and entity extraction

  • 04

    Consultation workflow and communication training

NEXT STEP

Need a different collection setting or annotation scheme?

An existing dataset can be licensed as it stands, or extended with additional speakers, scenarios and annotation layers around your study design.

Reply within one business day · NDA and data licensing supported · Dedicated academic track