Reading a project rubric: how to apply AI grading guidelines
How-to · Today
Every rating project comes with a rubric, and graders are checked against it rather than against their own opinion. How the two common rubric shapes work, the order to read a guidelines document in, and what to do when the rubric does not cover the case in front of you.
What a rubric is
On a grading project you are scored against the rubric, not against your own idea of a good answer. Mercor's glossary defines rubrics as "structured scoring guides used to evaluate model outputs", and says their job is to "ensure consistency across raters". Our guide to passing qualification tests makes the same point about tests: the guidelines are the rubric.
Rubrics come in two broad shapes.
A scale rubric gives you one rating along a defined ladder. Google's Search Quality Rater Guidelines use two: a Page Quality slider from Lowest to Highest, with in-between steps such as Low+ ("halfway between Low and Medium"), and a Needs Met scale from Fails to Meet up to Fully Meets.
A checklist rubric lists separate criteria, each met or not met. HealthBench, a medical benchmark published in May 2025, is an example: 262 physicians wrote 48,562 criteria across 5,000 conversations. Each criterion carries a point value between -10 and 10, and the negative criteria describe things a response should not do. A response's score is the points it earns divided by the maximum available, and it can drop below zero if it trips more negative criteria than it meets.
Read the guidelines in this order
- The scale or criteria definitions, especially the two ends. Know what the lowest rating requires before you read anything else.
- The worked examples. Google's guidelines call the examples "very important to illustrate how the concepts" should be applied, and they tell raters to examine them "to help calibrate your ratings". Score each example yourself before reading the answer.
- The stop conditions. Some answers end a task early: on Google's Page Quality tasks, a page that did not load or is in a foreign language gets no rating at all.
- The penalties. Find every negative criterion or automatic fail, since these move a score furthest.
Applying it, task by task
Judge each criterion on its own. HealthBench's grader goes through "each rubric criterion independently", and a human grader should do the same, because one strong first impression tends to colour every line after it.
Leave outside standards out. Google's guidelines say that unless the task indicates otherwise, ratings should rest "on the instructions and examples given in these guidelines", not personal opinion.
When the rubric is silent, say so. Google tells raters who meet a case the guidelines do not cover to rate it on their judgment and "use the comment section to explain your reasoning". On a paid project, ask in the official channel first; Outlier's Community Guidelines ask contributors to "use the appropriate channel for raising questions or concerns".
Re-read the guidelines whenever the project updates them, before your next task, since a rule you learned last week may have changed.
Where rubric work is listed
In October 2026, 102 of the 761 listings on this site named a rubric skill, 52 from Mercor and 50 from micro1. 57 of them list rubric writing, which means building rubrics, against 9 that list rubric application. Pay ran from $14 to $400 an hour, with a median midpoint of $75. micro1's Educator / Assessor / Writer pays $90 to $140 for rubric design and grading. Browse more under AI Safety & Evaluation.
Questions
- What is a rubric in AI training work?
- A structured scoring guide for judging model outputs. Mercor's glossary defines rubrics as structured scoring guides used to evaluate model outputs, there to keep different raters consistent. Some rubrics are a single scale, others are a checklist of criteria each marked met or not met.
- What should I do when the rubric does not cover a case?
- Ask through the project's official channel before guessing, and explain your reasoning in the comment or rationale field. Google's rater guidelines tell raters to rate a page by their judgment when the guidelines do not cover it and to use the comment section to explain why.
- Should I use my own judgment or the rubric?
- The rubric. Google's guidelines say ratings should be based on the instructions and examples in the guidelines, not on personal opinions or preferences. Your judgment comes in when you decide which part of the rubric applies.
Platforms covered here
Put this into practice
Every listing shows its pay and who it is open to.