Senior Software Engineer – Open Source, SWE-Bench Evaluation
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Software engineers reviewing open-source GitHub tasks, pull requests, and tests for Anyone AI. Evaluating reproducibility, quality, complexity, and benchmark suitability.
Responsibilities:
- Review software engineering tasks derived from real GitHub issues and pull requests
- Assess whether problem statements and success criteria are clear and complete
- Evaluate unit tests for correctness, coverage, and robustness
- Identify flaky tests, missing dependencies, version conflicts, and environment issues
- Determine whether tasks can be reliably reproduced across environments
- Assess the real-world difficulty and complexity of each task
- Provide clear recommendations on whether tasks should be accepted, improved, or excluded
- Review real-world bug fixes and feature implementations, repository setup, dependency management, multi-file and cross-module changes, and technical quality
Requirements:
- 3+ years of professional software engineering experience
- Strong experience working with large, multi-file codebases
- Experience reviewing pull requests, debugging issues, and maintaining production code
- Strong understanding of unit testing and test coverage
- Ability to evaluate whether tests correctly validate a solution without unnecessarily restricting implementation approaches
- Experience with dependency management, environment setup, and reproducibility
- Strong understanding of Git and GitHub-based development workflows
- Ability to analyze complex technical problems and provide clear written feedback
- Contributions to or maintenance of open-source projects (nice to have)
- Experience with SWE-Bench, SWE-Bench Verified, or similar coding benchmarks (nice to have)
- Experience with major Python open-source projects such as Django, Flask, scikit-learn, SymPy, matplotlib, requests, or pytest (nice to have)
- Experience with Docker, CI/CD, pip, conda, or dependency pinning (nice to have)
- Knowledge of test fixtures, test isolation, or property-based testing (nice to have)
- Experience designing technical assessments or reviewing coding challenges (nice to have)
- Experience with AI/ML evaluation, data curation, RLHF, or benchmark development (nice to have)
Benefits:
- Remote work
- Part-time, project-based consulting engagement
Similar Jobs
Software Engineering, Team Manager
Engineering Team Manager leading Pearson teams that support assessment services across testing, delivery and online proctoring. Strengthening people, delivery, quality and responsible AI practices.
Senior Software Engineer, Certificate Management
Senior Software Engineer leading PKI, cryptographic infrastructure, and certificate management at Coinbase. Building resilient services for critical internal and partner systems.
Tech Lead – Infrastructure and SAP Cloud Specialist
Tech Lead gerenciando Azure, AWS e segurança de infraestrutura para a MIGNOW. Apoio à conversão automatizada de SAP ECC para S/4HANA.
Senior Staff Platform Software Engineer – DevEx
Sr. Staff Engineer liderando plataforma, automação inteligente e DevEx no Grupo OLX, marketplace digital brasileiro. Evolução de portais internos, jornadas agênticas, métricas e FinOps.
Senior BI Technical Lead
Líder Técnico BI liderando equipes e soluções Power BI para a transformação digital do Grupo SysMap. Desenvolvendo modelos, dashboards e indicadores para diversos segmentos de negócio.
Junior Software Engineer, Java
Engenheiro de Software Java desenvolvendo e mantendo aplicações, APIs e testes no PagBank. Contribuindo para soluções que facilitam a vida financeira de pessoas e negócios.
Senior Software Engineer, Java
Engenheiro de Software Sênior desenvolvendo arquiteturas, APIs e microsserviços Java no PagBank. Contribuindo para produtos financeiros e a saúde da plataforma do banco digital.
Software Engineer – Build Infrastructure
Software Engineer owning Bazel build infrastructure for Glydways’ autonomous, carbon-neutral public transit system. Improving remote execution, CI/CD, toolchains, and developer workflows across the monorepo.
Senior Full-Stack Developer
Senior Full-Stack Developer building secure, scalable web applications for Associated Bank’s digital banking strategies. Leading projects, evaluating technologies, and mentoring developers.
Senior Fullstack Engineer
Senior Fullstack Engineer evolving enterprise web applications and integrations for CI&T, which transforms businesses through AI and technology solutions. Building scalable APIs, document processing, and SAP/Ivalua integrations.
Software Engineer, Product – Full-Stack: Angular, Kotlin, MongoDB
Software Engineer extending Vestlane’s digital KYC/AML platform for asset and fund managers. Building Angular/Kotlin features, MongoDB services, and public APIs end to end.
Senior Full-Stack Software Engineer
Engenheiro FullStack Sênior desenvolvendo a infraestrutura de crédito do agronegócio da AgRisk. Evolução de aplicações React, serviços Node.js e arquitetura distribuída ponta a ponta.
Staff Engineer, SAP ESPP
Staff Engineer supporting SAP S/4HANA eSPP environments for Nagarro, a digital product engineering company. Designing solutions, resolving complex bugs, and leading UAT and production rollouts.
Full Stack Java Engineer – Workforce Management
Full Stack Java Engineer building Java, React, and cloud-native applications for Miratech’s workforce management solutions. Delivering APIs, integrations, CI/CD pipelines, and production support.
Full Stack Java Engineer – Workforce Management, WFM
Full Stack Java Engineer building Java, React, and cloud-native applications for Miratech’s workforce management solutions. Developing APIs, microservices, integrations, and CI/CD pipelines.
Java Software Engineer
Java Software Engineer building Spring Boot microservices for a last-mile delivery technology company. Translating client needs into scalable backend solutions and implementation plans.
Senior Java Software Engineer – Angular
Senior Java and Angular engineer building scalable microservices and backend systems for Accesa, a Romanian technology company. Developing event-driven applications, CI/CD deployments, and database integrations.
Full-Stack Developer
Desenvolvedor Fullstack Pleno criando software com .NET, React e IA generativa. Especificando, testando e validando código e fluxos agentivos com LLMs.
Product Engineering Lead
Product Engineering Lead scaling Tether’s peer-to-peer Keet messaging and video app. Leading API and React frontend teams toward global consumer-product growth.
Product Engineering Lead
Product Engineering Lead building Keet, Tether’s peer-to-peer messaging and video calling app. Leading API and frontend teams toward globally scalable consumer features.