Software Engineer: LLM Evaluation & Repository Validation

What you'll do

Your job is quality control on the benchmark itself. You review and triage issues in open-source GitHub projects, prepare repositories for evaluation (containerisation and environment automation), and assess how thorough and well written the unit tests are. You then edit and run code locally to see how LLMs perform on bug-fixing tasks, help choose repositories and issues that are hard for models, and write up findings for researchers and reviewers.

Compared with the general Software Engineer listing, there is less task authoring and more environment setup and triage: the emphasis is on making real-world projects run reproducibly and deciding which issues are worth testing a model on. It is remote contract work with calibration and review sessions.

Who fits

• 3+ years of software engineering
• Strong experience in at least one of Python, C++, C#, Go, Rust or Ruby
• Proficiency with Git, Docker and basic pipeline setup
• Able to navigate complex codebases and run real-world projects locally
• Fluent written English

Preferred, not required: prior LLM research or evaluation projects, experience building or testing developer tools or automation agents, and open-source contribution or evaluation experience.

What it pays

The listing states $50–100 an hour.

Rex.Zone's listing states USD $50–100 per hour, "adjusted by country of residence". The ad does not say how the adjustment works, so the top of the range may not apply everywhere, and where an offer lands in the range is not published. Payment terms are not published on the listing.

Worth knowing

Good:

• Open worldwide, with no country restriction stated
• A wider language choice than the other software roles, including C#, Ruby and Rust
• Work that rewards environment and CI skills, not only algorithmic coding
• Clear task framing: triage, containerise, test, report

Less good:

• Pay is adjusted by country of residence and the ad does not explain how
• Verified-users-only applications and an entry exam come before any paid work
• The ad states no hours or volume, so how much work you get is unknown
• Contractor agreement terms on ownership and licence are still marked as to be confirmed by Rex Legal
• You must not submit export-controlled, military or defence-related technical data, or material you lack the right to share, and credentials and personal data must be stripped from anything you submit

About this listing

Posted on the Rex.Zone expert portal on 2 October 2026 and read from the signed-in portal on 29 October 2026. The requirements above are the ad's own; it lists no skills tags. Rex.Zone does not publish payment methods or payout timing, and the listing does not say how pay is adjusted by country. The portal marks applications as open to verified users only, and most roles sit behind an entry exam. See Rex.Zone.