What is reward model?
A model trained on human preference data to score answers, which then guides the main model during training.
A reward model is a separate AI model that scores answers the way human raters would. It is trained on preference data, then used during RLHF to score millions of answers the main model generates, far more than people could rate by hand.
Why trainers should know the term
When you rank answers on an RLHF project, your judgments are what the reward model learns. If raters reward long answers, the reward model learns that length is good, and the main model starts padding. This is why guidelines are strict about rewarding correctness and penalising filler.
Reward hacking
A model can learn to please the reward model without actually improving, for example by sounding confident or repeating the question. Evaluators who spot these patterns in model output help labs fix the reward model.
Example
After a training round, a model begins ending every answer with an offer to help further. Evaluators notice it, rate those endings as filler, and the next reward model stops rewarding them.
More terms: how models are trained
Full glossaryReady to look at roles?
Every listing shows its pay and who it is open to.