Senior Software Engineer: Python (LLM Evaluation & Repository Validation)

What you'll do

Python is the most common language in public SWE-style benchmarks, so this is the busiest strand of Turing's repository validation project. You help turn real repository histories into verifiable tasks that can test whether an LLM fixes bugs correctly.

The work:

• Analyse and triage GitHub issues across trending open-source Python libraries
• Configure repositories, including Dockerisation and environment setup
• Evaluate unit test coverage and quality
• Modify and run codebases locally to assess LLM performance on bug fixes
• Work with researchers to find issues that models find hard

You may also get the chance to lead a team of junior engineers.

Who fits

• 3 or more years of overall experience and strong Python
• Git, Docker and basic pipeline setup
• Comfortable navigating complex codebases and running real projects locally
• Open-source contribution or evaluation experience is a plus
• Nice to have: prior LLM evaluation work, or experience building developer tools or automation agents

What it pays

The ad does not publish a rate. Turing sets rates per project; see Turing.

Commitment and location

• 20, 30 or 40 hours a week, at least 4 hours a day, with 4 hours of overlap with PST
• Contractor assignment (no medical or paid leave)
• Contract length: 3 months, and the ad says the expected start is the following week
• Locations named: India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico

Worth knowing

Good:

• A stated 3 month term and quick start, which is more concrete than most sibling listings
• Practical work in a language most engineers already use daily
• Remote, with a choice of hours

Less good:

• No rate published
• Python is the most popular language for this work, so expect competition
• Pacific time overlap and country limits narrow who can take it
• Fixed term with no benefits or paid leave

How hiring works

Two interviews totalling about 75 minutes: 60 minutes technical, then 30 minutes of technical and cultural discussion.

About this listing

Read from Turing's job board on 18 October 2026. Of the five language variants in this repository validation set, Python and Rust are the two that state a 3 month contract. See Turing.