How QA reviewers score your AI training work
How-to · Today
Your work on an AI training platform is checked by reviewers, and the result decides whether you keep getting tasks. What Mercor, DataAnnotation and Outlier actually publish about quality review, what happens when a reviewer flags your work, and what none of them disclose.
What a reviewer checks
On review-based projects, a person checks your work before it counts. Mercor's glossary defines quality assurance as checking that contractors' work meets "the expected standards of accuracy, consistency, and completeness". In practice a reviewer looks at whether you followed the guidelines, whether the answer is right, and whether you treated similar cases the same way.
Mercor's time tracking and pay policy is the most specific public description of what fails review. It lists "deliverables that are off-scope or incomplete", "neglecting acceptance criteria", "recurrent rework resulting from overlooked instructions" and "broken links or assets that are non-runnable".
What happens when your work is flagged
Mercor publishes the sequence. A project lead or reviewer gives "concrete, actionable feedback tied to the requirements". You correct and resubmit, then the work is re-evaluated. If quality stays below expectations, Mercor "may end your assignment on the project". Its offboarding page adds that a project lead may first set a trial sprint with specific expectations and timelines.
Two details are worth knowing. Mercor states that low-quality output "does not invalidate legitimate time already logged", so hours stay payable even when work is rejected. And its project support FAQ says offboarding decisions are "conclusive": support cannot overturn them, though you can ask your point of contact for context.
DataAnnotation ties quality to which projects you see. Its FAQ says work availability depends on "your past performance on the platform" and that "subpar or problematic contributions may result in loss of access to future projects". It also says every pay tier includes higher rates "based on strong performance".
Outlier's Community Guidelines say breaking them can lead to "removal from tasks, disabled accounts, or other restrictions".
What nobody publishes
None of these three publishes, on its public pages, how a quality score is calculated, the threshold that triggers removal, or how many reviewers see a task. If a project shows you a score, ask in its official channel what the scale means and what counts as passing. Mercor's own advice is to "ask your project lead for feedback early".
Habits that keep a score up
- Read the acceptance criteria before you start, not after feedback.
- Write the rationale for a reviewer who has thirty seconds: the verdict first, then the evidence. Our guide to AI response evaluation shows where the rationale fits in a task.
- Treat feedback as a rule change. If a reviewer corrects one case, apply the correction to every similar case after it.
- Run anything that should run. Non-runnable output appears on Mercor's list of common quality issues.
QA is also a job
The reviewers are contractors too. In October 2026, 44 of the 761 listings on this site had QA, review or audit in the title, paying $15 to $350 an hour. Mercor's German Sonic Audit lists transcription audit and rubric application as its skills, at $41.50, and its SWE-Bench Task Auditor pays $70 to $90. More are listed under AI Safety & Evaluation.
Questions
- How is my quality score calculated?
- None of Mercor, DataAnnotation or Outlier publishes a formula on its public pages. What they do publish is what a low score leads to: fewer projects on DataAnnotation, feedback and possible removal from the project on Mercor, and removal from tasks or a disabled account on Outlier for breaking its guidelines.
- Do I lose pay if a reviewer marks my work as low quality?
- On Mercor's hourly projects, no: its pay policy says low-quality output does not invalidate legitimate time already logged, though quality affects whether you stay on the project. On per-task projects the terms differ, so check your contract.
- Can I appeal a removal from a project?
- Mercor says offboarding decisions made by the project team are conclusive and that support cannot overturn them, but you can ask your point of contact for context. Other platforms do not publish an appeal route on their public pages.
Platforms covered here
Put this into practice
Every listing shows its pay and who it is open to.