What is preference data?

Preference data is a collection of model answers paired with a human judgment of which one is better. Each record usually holds a prompt, two or more answers, the choice, and often a written reason and per-dimension scores.

How it is used

Preference data trains a reward model in RLHF, or trains the main model directly with methods such as DPO. Either way, the model learns to produce answers people prefer, which is why the quality and consistency of the judgments matter so much.

What makes good preference data

• Judgments follow the guidelines, not personal taste.
• Reasons name the specific difference, such as a wrong figure or a missing step.
• Ties are marked as ties instead of being forced into a choice.
• Close calls are flagged so reviewers can check them.

Example

Two answers to "How do I appeal a parking fine?" are both accurate. One gives the deadline and the form name; the other is vague. The preference record picks the first and gives the deadline as the reason.