What is Inter-annotator agreement?

Inter-annotator agreement measures how often different people give the same label to the same item. It is reported as a percentage or as a statistic such as Cohen's kappa, which corrects for agreement that would happen by chance.

Why projects track it

Low agreement means either the guidelines are unclear or some workers are not following them. Project leads use it to fix ambiguous instructions and to find annotators who need feedback. High agreement on hard items is also evidence that the dataset can be trusted.

What it means for you

Your own agreement with other annotators and with gold answers often feeds your quality score. Consistently disagreeing with the majority on clear cases can remove you from a project, while flagging genuinely ambiguous items is usually welcomed.

Example

Three annotators label 1,000 product reviews as positive, negative or neutral. They agree on 82 percent; most disagreements are on mixed reviews, so the guidelines are updated with a rule for reviews that praise one thing and criticise another.