Senior Software Engineer – AI Evaluation & Benchmarks, Python
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior Software Engineer contributing to AI model evaluation benchmarks and pipelines in a remote capacity. Requires expertise in Python and extensive software engineering experience.
Responsibilities:
- Design and build the coding benchmarks and evaluation pipelines used to test frontier AI models on real software engineering work:
- Design coding benchmarks that evaluate frontier models on real-world programming tasks — reasoning, debugging, and production-quality code
- Build and maintain scalable data pipelines for evaluation workflows
- Analyze model-generated code for correctness, reliability, and edge-case failures
- Construct structured evaluation scenarios across large repos and multi-language environments
- Provide detailed technical feedback on model performance and failure patterns
- Contribute to evaluation frameworks that set the bar for how coding ability is measured
- End result: benchmarks that meaningfully separate what frontier models can and can't do — and shape how the next generation is trained and improved.
- AI coding evaluation in one line: Design task → build harness → run model → analyze failures → feed findings back into the benchmark → evaluations that actually distinguish strong models from weak ones.
Requirements:
- 4+ years of professional software engineering experience (non-negotiable)
- Expert Python — clean, performant, well-tested code
- Hands-on experience working in large, complex codebases
- Proven experience designing and implementing LLM coding benchmarks and evaluation data pipelines
- Strong command of Git and modern development workflows
- Track record at a high-growth tech company or top-tier software organization
- Strong written English communication.
- Identity verification: Applicants must verify identity and have valid documentation to work as an independent contractor.
Benefits:
- Identity verification required for independent contractors in residence country
- Weekly payments via PayPal or Stripe
Similar Jobs
Senior Backend Engineer
Senior Backend Engineer building robust and scalable Integration Platform at Digibee. Contributing to architectural solutions and quality code development in a flexible remote work environment.
Senior FullStack PHP, Part Time
Engineer supporting multiple client-facing web development projects for a digital agency. Focus on PHP, JavaScript, CMS integration, and API services in a remote environment.
Senior FullStack PHP – Part Time
Senior engineer managing client-facing web development projects for a digital agency. Focusing on PHP, JavaScript, and CMS platform integrations in a remote setting.
Senior FullStack PHP, Part Time
Web Developer supporting multiple client-facing web development projects in a digital agency based in Colombia. Responsibilities include coding, feature development, and API integrations.
Senior FullStack PHP - Part Time
Engineer supporting multiple digital agency client web development projects. Involves PHP, JavaScript, and CMS integrations in a remote setting.
Senior Windows C++ Engineer
Senior Windows C++ Engineer managing software infrastructure for gaming tools. Building native infrastructure and ensuring successful application integration for 40M+ players.
Arquiteto de Soluções – Modernização, .NET, AWS
Software Architect leading the transformation of legacy applications to modern .NET architecture. Joining a financial technology company with over 28 years of market experience.
Desenvolvedor .Net Sênior
Senior .NET Developer designing and implementing solutions in .NET Framework and SQL Server at Extractta. Collaborating with cross-functional teams and enhancing software performance.
Full Stack Developer, Java / React – Senior
Senior Fullstack Java Developer specializing in robust and scalable applications with Extractta. Key responsibilities include modernizing legacy systems and implementing microservices architecture.
Senior Python Engineer
Senior Python Engineer designing scalable integration solutions for an AI startup. Collaborating with CTO and engineering team to enhance customer value with AI technologies.
Data Engineer, Python
Senior Data Engineer responsible for evolving user-facing experiences through AI-driven features. Building intelligent ecosystems and managing data pipelines in a high-impact, remote Agile team.
Senior Golang Developer
Senior Software Engineer at Qube Cinema designing and building technology platforms. Involves solving complex problems and mentoring junior engineers for high-quality software solutions.
Software Engineer – Java, Spring Boot
Mid-level Software Engineer in a Product Development Consultancy, building software solutions using Java and Spring Boot in a collaborative remote environment.
Programador/a Java, inglés
Java Developer with 5-10 years experience implementing business solutions at IRIUM. Requires high English proficiency and expertise in various Java technologies.
Senior Software Developer, Backend
Senior Software Developer focused on backend systems, modernizing components and collaborating with cross-functional teams. Must reside in Germany and have strong .NET or Java experience.
Lead Software Engineer, Backend
Lead Software Engineer at Varicent delivering scalable backend solutions for revenue management. Collaborating with global teams to enhance software performance and engineering standards.
Senior Software Engineer, Python, React
Senior Software Engineer assuring product quality at CESAR. Promoting team evolution and sharing technical knowledge in a relaxed work environment.
Software Backend Engineer
Backend Software Engineer designing and maintaining APIs and services for a fintech leader in fraud prevention. Remote-first culture with focus on collaboration and high engineering standards.
Senior Backend Software Engineer, Cloud Platform
Senior Backend Software Engineer developing scalable cloud platform solutions at Teleport. Focusing on Kubernetes, security, and analytics for AI-driven infrastructure.
Senior Backend Software Engineer – Cloud Platform Tools
Senior Backend Software Engineer working on cloud platform tools for Teleport. Focus on developing and releasing tools to enhance engineering productivity through open-source practices.