INSIGHTS
Annotation agreement: when κ = 0.8 is not enough
What a median of κ = 0.91 hides, when agreement coefficients mislead, and how stratified reporting replaces a single number.
In the acceptance sheet of an annotation project, Cohen’s κ is often the only agreement metric mentioned. κ = 0.8 is treated as the threshold for delivery: below it the batch goes back, above it the batch ships. But κ is a correlation coefficient, not a verdict. It can be inflated, and it can look respectable at the exact moment the annotation process deserves attention. The median inter-annotator agreement in our 2025 delivery batches was κ = 0.91, yet we never report that figure on its own.
What κ actually measures
κ estimates agreement between annotators after subtracting what chance alone would produce. Its denominator is the room left to improve beyond random labelling, which is why it becomes unstable when one label dominates the sample. Consider an extreme case: if 95% of segments contain no overlapping speech and both annotators tend to label “no overlap”, their raw agreement can be very high while κ stays modest. In the other direction, a rare category crossed by a single disagreement can pull κ down sharply.
This has two consequences. First, κ cannot be used alone to judge annotation quality. Second, κ cannot be compared across tasks. Character-level transcription accuracy, paralinguistic labels and dialogue-act classification have entirely different label spaces and boundary clarity, so a single threshold does not transfer between them.
For ordinal labels such as intensity, weighted κ is usually more informative than the unweighted version, because a one-step disagreement is not as costly as a four-step one.
Three ways κ goes wrong
In our reviews we keep running into three situations that push κ away from real quality:
- Severe class imbalance. A majority class covers more than 90% of samples, so κ is dominated by disagreement in the minority class and has a wide variance.
- Blurry boundaries. The annotation manual gives no decision rule for edge cases, each annotator develops a private standard, κ looks acceptable, and the errors cluster on the same segments.
- Annotators from the same source. Two annotators trained on the same examples with the same examples in front of them have their disagreement systematically smoothed away. κ is high, but reproducibility is not.
The first problem calls for Gwet’s AC1 or stratified reporting by class. The second requires going back to the manual to add boundary cases and adjudication records. The third requires independent annotators or third-party spot checks.
A fourth case is subtler: some annotation tasks have no single correct answer. Whether a sentence is ironic is a judgement different annotators can each justify. The goal there is not a high κ; it is to record the reasoning behind each label and to state the interpretive boundary of the tag set in the report.
Reference bands by task
Thresholds should be set per task rather than globally. The table below shows the internal bands we use. They are starting points drawn from past delivery batches, and every project still calibrates them against its own pilot annotation.
| Task type | Primary metric | Reference band | Note |
|---|---|---|---|
| Verbatim transcription | Character error rate | ≤ 3% | Against a reference transcript |
| Phoneme-level annotation | κ | 0.85 – 0.95 | Boundary phonemes spot-checked separately |
| Dialogue-act classification | κ | 0.75 – 0.90 | Many classes, fuzzy boundaries |
| Affect and paralanguage | Weighted κ | 0.70 – 0.85 | Weighted κ for ordinal labels |
| Event boundary detection | Agreement within tolerance | ≥ 90% | ±50 ms tolerance |
Bands matter because they tell the team where the target is before work begins, rather than leaving “is 0.8 high or low?” to the acceptance meeting.
Three things more useful than κ
An agreement report built on one number gives little guidance for improvement. We care about three things instead:
- Stratified agreement. Report by class, speaker and scenario to find the subsets where disagreement concentrates.
- Boundary case list. Collect every segment with a disagreement and turn the list into the basis for the next manual revision.
- Adjudication log. Every third-party decision is recorded with its reasoning, producing a traceable history of how the standard evolved.
Together these make a report that can explain why agreement is high or low, not just how high it is.
A workable agreement workflow
We usually proceed in the following order, with a concrete output at each step:
- Pilot annotation. Two annotators label 100 segments independently, compute initial agreement, and expose gaps in the manual.
- Manual revision. Add boundary definitions for disagreement cases and update the example library.
- Full double-blind annotation. The two passes stay invisible to each other to avoid influence.
- Adjudication. A third annotator resolves disagreements and records the reasoning; the adjudication rate is reported.
- Spot-check review. Add a 10% spot check on high-disagreement classes to confirm the revision worked.
- Delivery report. Provide stratified agreement, the boundary list and the adjudication log instead of a single κ.
Two operational details matter in practice. Annotators should not be able to see each other’s labels inside the tool, and annotation sessions should be long enough to keep focus but short enough to avoid fatigue. Agreement measured in the final hour of a long shift is not the agreement the corpus will show in production.
Closing
κ = 0.8 is not a universal safety line. It means something only within a particular task and category distribution. Annotation quality becomes controllable when agreement is treated as a phenomenon to be explained rather than a number to be reached. For researchers who reuse a corpus, a report that states boundaries and disagreements is worth more than a beautiful κ.
TOMCUBE
The TOMCUBE research team designs collection standards, annotation schemes and verification reports, and turns first-hand project experience into public method notes and benchmarks.
Related
Keep reading
More method notes on adjacent topics.
Research partnership
Turn your data questions into a verifiable design
If you are planning multimodal collection, annotation or validation, we can help you turn the research question into an executable protocol and a QA plan.
Reply within one business day · NDA and data licensing supported · Dedicated academic track