AI Agent Evaluator

What you'll do

AI agents are being sold on the promise that they will pay your bills and book your trips. This project checks that promise from the user's chair. You take two AI agents, side by side, and hand each the same real-life errand: a bill, a return, an email follow-up, a booking, a grocery order, travel planning, family logistics, an outing, a form. Then you watch what happens.

Each run follows the same loop:

• Start the scenario on your own phone, with your own accounts
• Approve the agent at key steps, or stop it when it goes wrong
• Screen record the whole attempt as evidence
• Score the run on completion, quality, control issues, satisfaction and time taken
• Write clear feedback on what the experience was actually like

micro1 budgets about 1.5 hours per run, and that figure includes the evidence and the scoring. The comparison is the interesting part: two agents doing the same job expose each other's weak spots quickly, and your notes on where you had to step in are likely to matter more than the final score.

Who fits

• Based in the US, with a bachelor's degree
• Someone who already uses AI tools in daily life, not just for the test
• 2–3 years of work experience, ideally in engineering, business management or office and operations work
• Careful about following instructions and documenting every step
• Clear in writing

Generalists are welcome, though the ad says domain experts are preferred. No coding or AI background is needed.

What you need to bring

This is a bring-your-own-setup job, which is worth checking before you apply:

• An active personal account on a popular AI assistant on a Free, Go or Plus plan (the ad specifically excludes Pro)
• A smartphone that can screen record

Because the errands run on your own accounts, think about what you are comfortable letting an agent touch. The ad does not say whether test purchases or bookings are reimbursed, or whether sandbox accounts are offered. Ask during the interview.

What it pays

$25–45/hour, micro1's published band. At about 1.5 hours per run, one run is worth roughly $37–68 across that range. Nothing in the ad explains what puts you at the top or the bottom of the band, so plan around $25 until you have an offer. Weekly hours and project length are not stated.

Timing

The project kicks off on Monday 12 October, with onboarding over the weekend before. The ad's own dates are slightly inconsistent (one line mentions Monday 10 October), and it was posted only days before the start, so later applicants may join a project already under way.

Worth knowing

Good:

• Work entirely from your phone, on your own schedule
• Getting paid for something many people already do for free
• A clear per-run structure with a published time estimate
• No coding or AI background needed
• 50 openings, flagged high demand

Less good:

• US only, and a bachelor's degree is required
• You use your own accounts and phone; privacy and any spending are your call, and reimbursement is not mentioned
• Screen recording every attempt means your notifications and personal data may end up on video unless you set the phone up carefully
• No hours, duration or pay structure published
• Contractor engagement: no benefits and your own tax to manage

How selection works

Apply, answer the screening questions, then a short interview (15–30 minutes). If your profile looks good, micro1 presents you to the client, and its Talent Ops team handles onboarding.

About this listing

Posted by micro1 on its own jobs portal on 9 October 2026 and read on 12 October 2026. The $25–45/hour band, 50 openings, US location, 1.5-hour run estimate and account requirements are its own published text. The client running the agents is not named. See micro1.