What is side-by-side (sxs) evaluation?
A task where you compare two AI responses to the same prompt, pick the better one and explain why. It is one of the most common forms of AI evaluation work.
Side-by-side evaluation, often shortened to SxS, shows you one prompt and two responses from AI models. You decide which response is better, usually on a scale (for example "A much better" to "B much better"), and write a short reason.
What you are judging
Most projects give you a rubric with several dimensions: whether the response follows the instructions, whether it is factually correct, whether it is safe, and how well it is written. You rate each dimension first, then give an overall preference. The ratings become preference data, which trains a reward model during RLHF.
How to do it well
Read the prompt twice before either response, so you know what a good answer needs. Check facts rather than trusting the more confident response. When both responses are flawed, the better one is the one with the less serious flaw, not the longer one. Your written reason matters as much as the choice: reviewers check that it names the specific difference.
Example
A user asks for a three-day itinerary. Response A gives four days in polished prose; Response B gives three days in plain bullet points. B is better, because A ignored a direct instruction.
Looking for this kind of work? AI Safety & Evaluation roles