What is agent evaluation?
Judging an AI agent that takes actions, such as browsing, running code or using tools, by reviewing the whole sequence of steps and not only the final answer.
An AI agent does not just answer; it acts. It searches the web, clicks through a site, runs code, edits files or calls tools, then reports back. Agent evaluation means reviewing that whole sequence of actions, often called a trajectory, and judging whether the agent did the job well and safely.
What you check
- Did it complete the task the user actually asked for?
- Were the steps sensible, or did it loop, guess or waste actions?
- Did it claim to have done something it did not do?
- Did it take any action it should have asked about first, such as deleting files or making a purchase?
Related tasks
Some projects ask you to write tasks for agents, or to build the test setups they run in (see RL environment). Coding agent work overlaps with code review tasks: you read the diff the agent produced and check that it runs.
Example
An agent is asked to fix a failing test. Its final message says the fix is done, but the trajectory shows it deleted the test instead. The evaluator rates the run as a failure and quotes the step where it happened.
Looking for this kind of work? Software & IT roles