How to write a good rationale when rating AI responses
How-to · Today
The rationale is the written reason beside your rating, and it is what a reviewer reads to decide whether your score holds up. How to write one that names the deciding issue and points to the evidence, with a weak and a strong example side by side.
What the rationale is for
On most rating tasks the score comes with a written reason, the AI response rating rationale, and that reason is what a reviewer reads to decide whether your score holds up. Our guide to AI response evaluation jobs covers the task as a whole. This page covers the writing.
Platforms say plainly that they want your own reasoning here. Mercor's LLM usage policy bars contractors from using ChatGPT or similar tools to evaluate model responses or to write justifications, because "the objective is to assess your own reasoning". Grammar fixes are allowed. DataAnnotation's software engineer role page puts the same point in five words: "your explanations are the training signal." Google's Search Quality Rater Guidelines, which Google publishes in full, give raters a comment box on every Page Quality task "to explain your rating".
Five habits that survive review
- Lead with the verdict. In the first sentence, say which response is better (or what score you gave) and the main reason.
- Point to the exact place. Quote the sentence, name the line of code, give the step number. "Contains an error" sends the reviewer hunting. "Step 3 divides by 4 instead of 52" does not.
- Use the rubric's words. If the dimension is called instruction following, call it that. Reviewers compare your score with the rubric, and your rationale should make that comparison easy.
- Rank the problems. A wrong fact that changes the answer matters more than a clumsy sentence. Say which issue drove the score.
- Keep your taste out of it. Google's guidelines tell raters that ratings "should not be based on your personal opinions, preferences, religious beliefs, or political views". "I prefer the shorter one" is a preference. "The user asked for three bullet points and B gives three" is a reason.
A weak and a strong rationale
This example is illustrative, written for this guide. The prompt asks what a $1,200 monthly rent comes to per week. Response A says $300. Response B says $276.92 and shows the working.
Weak:
B is better. It is more accurate and more detailed. A has some issues.
Strong:
B is better on accuracy, and accuracy decides this one. $1,200 a month is $14,400 a year, or $276.92 a week; A divides by four and overstates the weekly figure by about 8%. Both answer in the format asked for. B is longer because it shows the working, which helps here because the user can check it.
The strong version names the dimension that decided the comparison and shows the error with numbers. It also mentions what the two responses share, so the reviewer can see you checked both, and explains why the extra length does not count against B.
Why rationales get marked down
- They restate the score ("this is a 2 because it is poor").
- They praise polish and miss a wrong fact. Our guide on how to test AI models covers checking the claim rather than the tone.
- They contradict the score, for example listing only minor issues next to the lowest rating.
- They read as machine-written. Mercor says submissions "evidently produced by artificial intelligence will be identified and flagged".
Where this work is listed
In October 2026, 17 of the 761 listings on this site named written feedback as a skill and 57 named rubric writing, all from Mercor and micro1. Rationales also feed the comparison data described in our RLHF explainer. Current roles are under AI Safety & Evaluation.
Questions
- How long should a rationale for an AI response rating be?
- Long enough to name the issue that decided the score and point to where it is in the response. Two to four sentences covers most comparisons. If the project sets its own length rule, follow that instead.
- Can I use ChatGPT to write my rationale?
- Not on Mercor. Its LLM usage policy forbids using language models to evaluate model responses or to write justifications, and allows them only for grammar and light rephrasing of your own ideas. Other platforms set their own rules, so check the project instructions.
- What does a reviewer look for in a rationale?
- Whether it supports the score you gave, whether it points to specific evidence in the response, and whether it uses the rubric's own criteria. A rationale that only restates the score in words gives the reviewer nothing to check.
Platforms covered here
Put this into practice
Every listing shows its pay and who it is open to.