What is consensus labeling?

Consensus labeling means giving the same item to several people independently and keeping the label most of them chose. It is a simple way to get reliable labels from workers who each make occasional mistakes.

How it works

An item might go to three or five annotators. If they agree, the label is accepted. If they split, the item may go to more annotators, to a senior reviewer, or be flagged as genuinely ambiguous. Some systems weight each vote by the worker's track record.

What it means for your work

• Your label is compared with other people's, so consistency with the guidelines matters more than originality.
• Often disagreeing with the majority on clear items can lower your quality score.
• On ambiguous items, flagging the ambiguity is usually more useful than forcing a choice.

Consensus is closely tied to inter-annotator agreement, which measures how often annotators agree overall.

Example

Five annotators label a photo as "dog", "dog", "dog", "wolf" and "dog". The consensus label is dog. A second photo splits three to two, so it goes to a reviewer, who adds a note to the guidelines about husky breeds.