What is DPO?

DPO, short for Direct Preference Optimization, is a way to train a model on preference data in a single step. Classic RLHF first trains a separate reward model and then uses it to tune the main model. DPO skips the reward model and adjusts the main model directly so that preferred answers become more likely and rejected ones less likely.

Why it matters to trainers

For the person doing the rating, nothing changes: you still see a prompt and two answers, pick the better one and explain why. What changes is how directly your judgment shapes the model. With no reward model in between, inconsistent or careless choices pass straight into training, which is why guidelines stress consistency.

Variants

Several related methods have followed DPO, with names such as IPO, KTO and ORPO. Some accept a simple thumbs up or down instead of a pair, which is why some projects ask you to rate single answers rather than compare two.

Example

A project collects 20,000 pairs of answers to coding questions, each marked with the better one. The lab trains on them with DPO overnight, with no reward model step, and asks the same raters to spot-check the new model's answers the next week.