Software Engineer (Developer Workflow Evaluation)

What you'll do

This is QA for AI developer agents. You work through structured assignments that mimic a real developer's day (branching, opening and reviewing pull requests, triaging issues, running CI workflows) using AI-powered tools, then grade whether the tool actually got it right. Grading is hands-on: you verify outputs in the real environment rather than judging from the transcript.

A good share of the work is building the stage. You seed and maintain reproducible test environments, such as repositories with genuine commit histories and working CI, and reset them between runs. You configure and troubleshoot integrations and authentication flows across several software services, and you document product behaviour that nobody has written down, by testing it until you know.

Calibration sessions keep grading consistent across the contributor group.

Who fits

• At least three years of professional software engineering with strong Git/GitHub habits: pull requests, review, branching, reading diffs, diagnosing CI logs
• CI/CD fluency, ideally GitHub Actions, including writing custom workflows and making builds reproducible
• Working knowledge of connectors in Claude and ChatGPT
• Solid Python or Bash scripting for environment setup and reset, plus REST APIs, OAuth and webhooks
• Familiarity with enterprise SaaS such as Slack, Google Workspace or Microsoft 365; workspace admin experience helps
• Everyday use of AI assistants or agents in development work; agent tool-calling or Model Context Protocol experience is a plus
• A QA, evaluation or annotation background with consistent rubric use
• Excellent written and spoken English and disciplined process-following

What it pays

$50–70/hour, micro1's published band, fairly narrow at $20 between floor and ceiling. The ad publishes no pay mechanism, weekly hours or project length.

Worth knowing

Good:

• Unusual, practical work: testing how AI agents handle real repos and SaaS integrations
• Rewards the unglamorous skills (CI debugging, OAuth plumbing, environment hygiene) that most AI training ads ignore
• A predictable band
• Relevant experience for anyone moving toward agent tooling or developer productivity roles

Less good:

• Only four openings, and the listing is about a month old
• A long, specific skills list; SaaS admin and connector experience narrow the field
• Environment resets and auth troubleshooting can eat time the grading does not reflect
• No hours, duration or pay mechanism stated
• Contractor engagement: no benefits and your own tax to manage

About this listing

Posted by micro1 on its own jobs portal about a month before we read it on 24 September 2026. The $50–70/hour band, four openings and three-year experience preference are its published text. micro1 lists several differently scoped Software Engineer roles; this one is the evaluation-and-tooling variant rather than a task-authoring one. No country restriction is stated and the customer is not named. See micro1.