Skip to content
Labeling Jobs

What is instruction following?

Published 

Whether an AI response does exactly what the prompt asked, including format, length and constraints. Usually the first dimension rated in an evaluation task.

Instruction following measures how closely a response obeys the prompt. It covers the explicit requests ("three bullet points", "under 100 words", "in Spanish", "do not mention prices") and the implied ones, such as answering the question that was actually asked.

Why raters check it first

A response can be accurate and well written and still fail the user, because it ignored a constraint. Most rubrics rate instruction following as its own dimension, and many projects treat a broken explicit instruction as a major error regardless of how good the rest is.

How to rate it

Before reading the response, list every instruction in the prompt, including the small ones. Then check each one. Count words or items when the prompt sets a number; do not estimate. When a prompt's instructions conflict, check the guidelines for which one wins.

Example

The prompt asks for a product description in under 50 words with no exclamation marks. The response is 48 words and has one exclamation mark. In a side-by-side evaluation, it loses to a plainer response that follows both rules.

More terms: quality and review

Full glossary