Software Engineer – AI Evaluation
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Software engineers reviewing realistic programming tasks and evaluation harnesses. Providing expert feedback to Terac, a platform connecting AI researchers with vetted human experts.
Responsibilities:
- Review and assess the quality of realistic programming tasks
- Evaluate coding environments and technical harnesses for AI testing
- Walk through potential flaws in provided code structures
- Provide feedback on the difficulty and realism of coding challenges
- Assess the technical accuracy, complexity, and realism of coding challenges
- Explain the logic of evaluation environments
- Compare evaluation environments with standard industry practices
Requirements:
- Professional experience as a software engineer or developer
- Familiarity with building or verifying programming tasks
- Experience with code review and evaluation harnesses
- Comfortable discussing technical architecture and testing methodologies
- Hands-on experience building, reviewing, or testing realistic programming tasks
- Familiarity with evaluation harnesses
- Open to full-stack developers, backend engineers, test automation engineers, and systems architects
Benefits:
- Paid interview/study participation
- $75 per hour compensation
Similar Jobs
GTM AI Engineer – Deal Desk
GTM AI Engineer building AI agents and workflows for Motive’s fleet-operations platform. Automating Deal Desk processes and scaling secure internal tools for Sales and GTM teams.
AI Engineer
AI Engineer building scalable AI, cloud, and cybersecurity solutions for Echelon Services’ federal and commercial clients. Developing applications, automation workflows, and MVPs using Go and Python.
Senior Staff AI Engineer
Senior Staff AI Engineer researching autonomous agents and LLMs at iFood. Building scalable AI prototypes and validating emerging technologies for production impact.
Founding SDR Lead – AI-powered Lifestyle Coaching
Founding SDR Lead building outbound sales for FitMate’s AI-powered lifestyle coaching platform. Creating prospecting systems, booking qualified meetings, and managing pipeline through decisions.
AI Engineer
Contract AI Engineer creating MCP-based reinforcement learning environments for next-generation AI systems. Designing deterministic tests and reference solutions for complex software engineering tasks.
Senior AI Engineer – GTM
Senior AI Engineer building production AI systems for Paddle, a Merchant of Record for digital product companies. Automating GTM workflows across segmentation, account research and sales operations.
Senior Cloud, Data & AI Engineer
Senior Cloud, Data & AI Engineer leading secure AWS, Celonis, and agentic AI solutions. Delivering reusable cloud products for federal agencies and financial-control modernization.
Senior Java Developer – AI Development
Senior Java Developer using Java, Spring Boot, and Claude Code on production projects. Designing, reviewing, debugging, and validating AI-generated software remotely.
Director, AI Architecture – B2C
BMO banking AI architecture director shaping customer experiences, personalization, and digital banking. Leading enterprise AI platforms, Responsible AI governance, and transformation across business units.
Director, AI Architecture – B2C
AI architecture director advancing BMO’s enterprise banking platforms and customer experiences. Leading responsible GenAI, cloud architecture, governance, and transformation across business units.
Senior Python Developer – AI, Data Integration
Senior Python Developer building RAG pipelines, autonomous agents, and MCP integrations for SOFTSWISS’s AI ecosystem. Designing scalable AI services with Python, distributed systems, and enterprise data.
AI Engineer – .NET, Azure AI
AI Engineer building .NET/C# backends and Azure/OpenAI solutions for Imaginary Cloud, a software development and UX/UI design company. Deploying generative AI products for global clients.
Senior AWS AI Engineer
Senior AWS AI Engineer designing production-grade Bedrock and RAG solutions. Helping Keyrus industrialise AI for organisations through cloud engineering and client delivery.
Senior Director – AI Engineering, Applied AI
AI engineering leader advancing FICO's global analytics software platform with LLM-powered fraud detection and decision automation. Leading teams, model governance, monitoring, and scalable responsible AI systems.
Applied AI Engineer
Applied AI Engineer building AI-powered internal tools for BLP Digital, an ERP automation SaaS company. Shipping LLM, agent, and workflow solutions across hiring, finance, and operations.
ServiceNow AI Developer
ServiceNow AI Developer building AI-driven applications and workflows for enterprise customers. Integrating machine learning, NLP, and predictive analytics across ServiceNow modules.
Senior Staff Engineer – AI Innovation Lead, Elixir, Ruby, Flutter
Senior Staff Engineer leading AI adoption, technical standards, and cloud architecture. Delivering custom software projects for SmartLogic’s consultancy clients.
Senior AI Engineer
Senior AI Engineer building MCP integrations and production AI agents for Charger Logistics’ transportation operations. Automating dispatch, billing, compliance, and fleet workflows.
Software Engineer, AI Solutions
Software Engineer building advanced Conversational AI solutions for NICE’s global customer experience, financial crime, and public safety software platform. Leading enterprise implementations, technical consulting, training, and integrations.
Líder Técnico de IA, Arquitectura, Preventa
Líder Técnico de IA y arquitectura para knowmad mood, compañía de transformación digital. Dirigiendo soluciones empresariales, preventa estratégica, APIs, cloud e innovación con IA generativa.