Senior Software Engineer: LLM Evaluation & Repository Validation

What you'll do

The project builds verifiable software engineering tasks from the histories of public repositories, with a human in the loop, to train and evaluate LLMs on realistic problems. Your side is the repository work. You analyze and triage GitHub issues across trending open-source libraries, set up and configure repositories (including Dockerization and environment setup), evaluate unit test coverage and quality, and modify and run codebases locally to see how an LLM performs at bug fixing. You also help researchers pick repositories and issues that are hard for models, and the ad mentions chances to lead a team of junior engineers.

The ad wants tech-lead-level experience with well-maintained, widely used repos (500+ stars).

Who fits

• Strong experience in at least one of Python, JavaScript, Java, Go, Rust, C/C++, C# or Ruby
• Proficiency with Git, Docker and basic pipeline setup
• Ability to navigate complex codebases and run, modify and test real projects locally
• Open-source contribution or evaluation experience is a plus
• Nice to have: prior LLM research or evaluation work, and experience building or testing developer tools or automation agents

What it pays

The ad does not publish a rate. Turing sets rates per project; see Turing.

Worth knowing

Good:

• Practical, hands-on work that suits engineers who like environments and tooling more than prose writing
• Fixed shape: 20 hours a week for 3 months
• Remote, with a path to leading junior engineers

Less good:

• Some overlap with PST is required
• Contractor assignment with no medical or paid leave
• No pay figure
• The "tech lead level" and 500+ star familiarity bar is steep

The ad does not describe an assessment. Turing's help centre mentions skill tests and, for some roles, a live interview.

About this listing

Read from Turing's job board on 18 October 2026; first posted in May 2025. The project description and terms are Turing's; the commentary is ours. See Turing.