INSIGHTS
Synchronization precision: the first quality line multimodal corpora overlook
What a 1.4 ms sync error actually means, where it comes from, how to measure it, and what an auditable synchronization report should contain.
In acceptance meetings, research teams look first at corpus size, transcription accuracy and annotator agreement; synchronization precision tends to come last. Yet when the audio, video, eye-tracking and physiological streams of the same conversation are placed on a single timeline, synchronization is the first thing that fails. A 40 ms offset is enough to invalidate a conclusion such as “the participant’s pupil dilated 200 ms after hearing the keyword”.
Across our 2025 delivery batches, the median end-to-end synchronization error was 1.4 ms. That number is not a specification printed on a device. It is measured segment by segment, from capture through encoding and alignment to delivery. This article explains where the error comes from, how to quantify it, and what an auditable synchronization report should contain.
Where synchronization error comes from
Every stream in a multimodal session runs on its own clock. Cameras follow a frame rate, microphones follow a sample rate, and eye trackers and physiological amplifiers each carry a different internal oscillator. As long as those clocks are not constrained by a shared external time base, error accumulates linearly with time. In a 40-minute session, a 0.02% difference in drift between two devices is enough to produce nearly 0.5 seconds of offset by the end.
In practice, the error originates in four places:
- Device clocks. Oscillator accuracy varies from roughly 10 ppm to 100 ppm across vendors, and long recordings always drift.
- Software buffering. Capture buffers, encoding queues and disk-write strategies introduce delays that are not constant.
- Triggering. Starting several programs with a software button leaves tens of milliseconds between their actual start times.
- Post-processing. Transcoding, downsampling and segment concatenation reintroduce error whenever time stamps are not preserved.
The first three can be controlled at the point of capture. The fourth can only be contained by a rule: never overwrite raw data.
Vendor specifications are a poor proxy for what actually reaches delivery. A camera may claim frame-level accuracy while its recording software stamps frames when they are written to disk rather than when the sensor is read out. Two devices can both be accurate and still drift apart, because accuracy and stability are different properties. An audit has to measure the combination, on the actual files produced in the actual session.
Turning error into measurable metrics
A synchronization claim is worthless without numbers attached to it. We measure the following metrics and report them for every delivery batch.
| Metric | Definition | Delivery threshold |
|---|---|---|
| Initial offset | Time difference between the first valid event in each stream | ≤ 2 ms |
| Clock drift | Relative change in accumulated offset per unit time | ≤ 0.02% |
| Frame-level jitter | Standard deviation of intervals between adjacent frames | ≤ 1 ms |
| Median end-to-end error | Median of segment-level checks in delivered files | ≤ 2 ms |
| Sync failure rate | Share of checked segments outside the threshold | ≤ 0.5% |
A median of 1.4 ms means that in 30 fps video, the distance between an audio event and the corresponding picture is under half a frame. For onset detection, turn-taking and emotion annotation, that is already below the resolution of human judgment.
Six controls at the point of capture
Synchronization is not fixed in post; it is protected on site. We run the same six actions for every project:
- One external time base. Lock every device to the same PTP or Genlock source instead of letting each one keep its own time.
- Hardware triggering. Start recorders from a shared sync signal, not from a person clicking a button.
- An opening sync rehearsal. Record 60 seconds before each session, including one hand clap and one flash, as a per-batch alignment anchor.
- Drift monitoring. Insert a sync marker every 10 minutes during long sessions so the recording can be corrected in segments.
- Read-only raw data. Every transcode or process writes a new file; originals keep their timecode and device metadata.
- Automatic metadata logging. Device model, firmware version, sample rate and time-base source go into the capture log and ship with the delivery.
These actions add 15 to 20 minutes of preparation per session. Compared with the cost of re-collecting a batch, it is the cheapest investment in the project.
One detail is easy to miss: the sync markers themselves need a definition. The acoustic onset of a hand clap and the frame where the palms meet must be annotated with the same rule, or calibration results will vary from one annotator to the next. Our reports state the anchor definition explicitly so that batches remain comparable.
What a delivery report should contain
An auditable synchronization report has at least five parts: the device and firmware inventory, a description of the time-base design, the anchor and spot-check records, per-segment error statistics, and a note on any known deviation. The last item is often omitted, yet it matters most to researchers who will reuse the data. If one segment carries a 12 ms offset because a device restarted, saying so is more useful than hiding it inside an average.
One practical habit is to keep a reference session per device configuration. Re-run the same 60-second rehearsal whenever firmware, cabling or capture software changes, and compare the measured offset with the previous baseline. Most sync failures show up here first, long before they reach a delivery batch.
Synchronization precision matters not because of the number itself, but because it determines which questions a corpus can answer. Once error is stable at the millisecond level, researchers can return to the research question instead of repeatedly suspecting the equipment.
TOMCUBE
The TOMCUBE research team designs collection standards, annotation schemes and verification reports, and turns first-hand project experience into public method notes and benchmarks.
Related
Keep reading
More method notes on adjacent topics.
Research partnership
Turn your data questions into a verifiable design
If you are planning multimodal collection, annotation or validation, we can help you turn the research question into an executable protocol and a QA plan.
Reply within one business day · NDA and data licensing supported · Dedicated academic track