What is semantic segmentation?

Semantic segmentation is labeling every pixel of an image with the class it belongs to, such as road, pavement, car, person or sky. Where a bounding box gives a rough rectangle, segmentation gives an exact outline, which models need for tasks like self-driving, medical imaging and photo editing.

What the work looks like

You trace the outline of each region with a polygon or brush tool, or correct a mask that a model has drafted. Guidelines define the classes and the hard cases: where a road ends and a kerb begins, whether a person's bag belongs to the person, how to treat shadows and reflections. Instance segmentation goes one step further and separates each object of the same class, so two people get two separate masks.

Why it pays more than boxes

Segmentation is slower and needs a steadier hand and more judgment. Medical segmentation, such as outlining tumours in scans, is usually reserved for clinicians or trained specialists.

Example

In a dashcam frame, the annotator corrects a pre-drawn mask where the model has merged a cyclist into the road, splits them into separate regions, and fixes the edge of the pavement that was cut short.