Software Engineer: AI Code Evaluation & Benchmarking (US candidates only)

What you'll do

You act as a code reviewer for frontier models: you read what an AI wrote, decide whether it is actually correct, and build the datasets and rubrics that let others measure the same thing consistently.

• Review AI-generated code for correctness, efficiency, maintainability and fit with requirements
• Check whether proposed solutions to engineering tasks meet the expected outcome
• Debug code, reproduce issues and verify fixes in different environments
• Judge model explanations and reasoning for technical accuracy
• Create, refine and maintain evaluation datasets, benchmarks and grading rubrics
• Find edge cases and failure modes where models struggle
• Document findings and agree quality standards with the project team

Who fits

• Bachelor's or Master's degree in computer science, software engineering or a related field
• 3 or more years of professional software engineering
• Strong in one or more of Python, Java, C/C++, Go, Swift, Objective-C, PHP or SQL
• Solid grasp of data structures, algorithms and design principles, with code review experience on production codebases
• Git and modern workflow familiarity, and strong written communication
• Plus: AI/ML annotation, NLP, prompt engineering or LLM project experience. Highly preferred: experience evaluating AI-generated code or building benchmarks

What it pays

The ad does not publish a rate. Turing sets rates per project; see Turing.

Commitment and location

• At least 4 hours a day and 20 hours a week, with 4 hours of overlap with PST
• Contractor assignment (no medical or paid leave)
• Contract length: 1 month, with the expected start the following week
• Location: US only

Worth knowing

Good:

• A wide language list, including less common ones like Swift, Objective-C and PHP
• A fast start
• The screening is a coding test, not a panel interview

Less good:

• No rate published
• One month only, with no extension mentioned
• US residents only
• The listing is filed under Turing's "Legal & Law Services" category, which does not match the engineering content (probably a board tagging error)

How hiring works

The ad describes an online automated coding challenge covering Python and a Docker test, and refers to it as RLHF-related.

About this listing

Read from Turing's job board on 18 October 2026. See Turing.