Skip to main content

SCHEME · TRAIN · REVIEW

Annotation management

Start from the label set and annotation manual, train annotators, run double-blind annotation with adjudication, and deliver annotation data with an agreement report.

Deliverables

  • Annotation manual and label set
  • Annotator training and certification records
  • Double-blind annotation and adjudication results
  • Agreement report with κ / α and confidence intervals
Service module
03 / 05
Deliverables
4 items
Process stages
4 steps
Quality verification
Batch-level

01 / Service scope

What we actually do

Each capability can be switched on as a standalone module or combined into a full delivery chain.

01

Label set and manual

Define annotation hierarchy, granularity, mutual-exclusion rules and boundary examples.

02

Annotator training

Standardized training followed by a certification test; only passing annotators join production batches.

03

Double-blind and adjudication

Two annotators work independently, and disputed samples go to a third-party adjudicator.

04

Agreement monitoring

Compute κ or α per label dimension to surface ambiguous boundaries early.

05

Versioning and sampling

Every annotation file carries a version; stratified sampling runs before delivery.

02 / Workflow

How it moves forward

Every stage has defined outputs and checkpoints, so progress and risk surface during the work rather than at acceptance.

  1. 01W1

    Label set design

    Agree on hierarchy, granularity and decision rules with the research team.

  2. 02W2

    Trial annotation and revision

    Annotate 30–50 samples to expose boundary cases and revise the manual.

  3. 03W3–W8

    Training and production

    Annotators pass the agreement test, work double-blind, and disputes go to adjudication.

  4. 04W9

    Review and report

    Run stratified sampling and issue the agreement report.

03 / Service notes

Who it is for, and how to begin

What we do

Annotation determines how fine a question the data can answer. We start with the label set: together with the research team we define annotation levels, boundary rules and disagreement handling, and write a manual that can be trained against, reviewed and versioned. Annotators work from that manual, not from personal intuition.

Execution runs in three steps. First, a trial annotation on a small sample exposes ambiguous boundaries and forces a revision. Second, training and certification: annotators must pass an agreement test before they join a production batch. Third, double-blind annotation with third-party adjudication, so disagreements enter a review queue instead of being averaged away. Every sample records the annotator, timestamp and version.

The manual is the real deliverable underneath the labels. It includes positive and negative examples, tie-break rules for overlapping categories, and a change log so annotators can see why a rule changed. When a scheme is revised mid-project, earlier samples are re-coded where the change is material, and the version history records which files were affected.

Agreement is not a number computed at the end. We build Cohen’s κ and Krippendorff’s α into the trial stage, sample weekly during production, and pause to revise the manual when one label class keeps splitting. Finding a problem in week two is cheaper than discovering a manual flaw across the whole batch at delivery.

Who it’s for

Teams that need structured labels: transcription and prosodic annotation, classroom discourse function coding, emotion and paralinguistic labels, clinical variable annotation for patient–clinician dialogue, eye-tracking area-of-interest segmentation, and AI teams preparing high-quality supervision signals for model training. We also take over existing schemes when a second opinion on their reliability is needed. Beyond the common tasks, we build schemes for gesture and co-speech annotation, second-language error coding and music performance events. Where no established convention exists, we prototype the scheme on a small sample and report its reliability before committing the full batch.

How it works

  1. Label set design. Agree on hierarchy, granularity, mutual-exclusion rules and examples.
  2. Trial and revision. Annotate 30–50 samples to surface boundary cases, then revise the manual.
  3. Training and production. Annotators pass the agreement test, work double-blind, and send disagreements to adjudication.
  4. Review and report. Run stratified sampling and issue the agreement report.

Capacity is planned from the trial stage: if the sample shows a high disagreement rate, we budget extra adjudication time rather than assuming annotators will speed up.

Delivery and verification

Deliverables include the annotation manual and label set, training and certification records, double-blind annotation and adjudication results, and the agreement report (Cohen’s κ or Krippendorff’s α). We report agreement per label dimension instead of a single aggregate score; low-agreement labels come with a stated cause and a revision recommendation. All annotation files carry version numbers and change logs. If the client has an existing acceptance threshold, we write it into the test plan and check it item by item. Delivery includes a label dictionary with definitions and value ranges, annotator IDs mapped to a separate key, and a summary of which samples were adjudicated and why.

How to get started

Prepare a representative sample and a first draft of the labels. We estimate annotation effort and reliability risk, then quote timeline, staffing and price. Projects with an existing annotation system can begin with a reliability audit before deciding whether a rebuild is warranted. If a project has a hard deadline, tell us the date early: reliability work does not compress like annotation throughput does.

04 / Deliverables

What you receive

Deliverables are organized to be citable and reproducible: naming, metadata and verification records are complete, ready for analysis or archiving.

Deliverable 01

Annotation manual and label set

Deliverable 02

Annotator training and certification records

Deliverable 03

Double-blind annotation and adjudication results

Deliverable 04

Agreement report with κ / α and confidence intervals

05 / FAQ

Common questions

If your question is not covered, contact us directly.

What agreement level counts as acceptable?

We agree thresholds in advance based on task and label complexity: exploratory annotation usually targets κ ≥ 0.70, while citable delivery typically requires κ ≥ 0.80. If the threshold is missed, we revise the manual rather than ship as-is.

Do annotators understand the domain?

Annotators are matched to the project by background in linguistics, clinical work or education, and receive domain training and a term glossary.

Can you use our existing annotation scheme?

Yes. We run a reliability audit first, confirm the rules are workable, then take on annotation and add boundary clauses and examples where needed.

How is annotation data delivered?

As TextGrid, ELAN, JSON or CSV, with a label dictionary, annotator IDs and version history.

Start a project

Design the annotation system before annotating

Bring a representative sample and a first draft of the labels. We assess annotation effort and agreement risk, then quote timeline and staffing.

Reply within one business day · NDA and data licensing supported · Dedicated academic track