Task Development Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Task Development Engineer creating difficult AI evaluation tasks for METR, a nonprofit assessing AI capabilities and risks. Improving task quality, scoring, and evaluation infrastructure.
Responsibilities:
- Develop difficult, novel tasks for AI models that remain challenging as model time horizons grow
- Perform quality assurance for existing tasks, verifying solvability and that models receive only the necessary information
- Baseline tasks within areas of expertise when helpful
- Score task completions from AIs or human baseliners
- Improve task development infrastructure and workflows
- Contribute to evaluations of frontier AI systems supporting METR’s Time Horizons methodology
Requirements:
- Several years of experience working on complex software engineering projects and codebases
- Experience building hard, ideally agent-based, AI evaluations
- Experience with evaluations such as RE-Bench, HCAST, SWE-bench Verified, Cybench, or GPQA
- Ideally experience using the Inspect framework
- High attention to detail, including spotting misspecifications and ambiguity
- Familiarity with METR infrastructure, including Hawk, is a plus
- Familiarity with the methodology behind METR’s Time Horizons work is a plus
- Availability for 20–40 hours per week
- Minimum of 1 hour of overlap with the Pacific Coast Time workday
Benefits:
- Flexible schedule determined by you
- Remote work worldwide
- Flexible working hours of 20–40 hours per week
- Individuals who contribute >80 hours will be acknowledged in the final research output (if desired)
Similar Jobs
Framework Engineer
Framework Engineer expanding Oneleet’s cybersecurity and compliance platform. Mapping frameworks to controls, automating updates, and improving compliance workflows for customers.
Forward Deployed Engineer
Senior Forward Deployed Engineer building Palantir Foundry healthcare solutions for Endava. Architecting data pipelines and applications supporting kidney care and operational transformation.
Build Engineer
Build Engineer improving CI/CD, release systems, and automation for Simformotion’s Cat Simulators software. Supporting reliable delivery across multiple products and engineering platforms.
Kafka Engineer, Streaming
Kafka and streaming engineer developing Java, Flink, and Confluent solutions for client integrations. Designing, deploying, testing, and supporting event-driven applications.
Senior Process Control Engineer – Fiberline
Senior Process Control Engineer improving fiberline control systems for Georgia-Pacific pulp and paper mills. Troubleshooting DCS operations, optimizing assets, and supporting capital projects across US sites.
Process Controls Engineer – Corrugated
Process controls engineer optimizing corrugated manufacturing lines for Georgia-Pacific. Modifying PLC/HMI strategies, commissioning equipment, and improving reliability across West Coast sites.
Federal Process Engineer
Federal Process Engineer modernizing Department of Defense and Energy facilities for Salas O’Brien. Designing, automating, and improving industrial processes while supporting safety, commissioning, and multidisciplinary delivery.
Low Code Engineer – OutSystems, Mendix
Low Code Engineer developing robust, scalable applications with OutSystems for Kyndryl. Collaborating with teams and utilizing modern frameworks for high-performance solutions.
Lift Engineer
Lift Service Engineer at Stannah providing lift servicing and maintenance across Bristol and surrounding areas. Joining a family-owned company providing accessibility solutions since 1867.
Installation Engineer
Installation Engineer for AirStrip, focusing on software installations and Tier 2 support for healthcare clients. Ensuring effective product operation and customer satisfaction through technical issues resolution.
Installation Engineer
Installation Engineer for AirStrip completing installations, configurations, and testing of software for healthcare clients. Engaging with customers and providing tier 2 technical support.
Manufacturing Engineer – Transition Standard Work
Manufacturing Engineer supporting Work Transitions at Pratt & Whitney, ensuring product quality and operational efficiency. Collaborating with teams to manage transitions and risks in manufacturing processes.
OpenStack Engineer, English
OpenStack Engineer at VEXXHOST designing, deploying, and supporting large-scale OpenStack environments. Collaborate with global clients and internal teams to resolve technical challenges and optimize infrastructure.
Graduate Engineer Trainee
Graduate Engineer Trainee managing resources and logistics for Multi-Modality Imaging Equipment maintenance at customer sites in India. Engaging in troubleshooting and customer relationship management.
Senior Engineer, Telephony
Telephony Engineer specializing in Five9 solutions for communication infrastructure at Ensemble. Engage in system design, optimization, and collaboration with cross-functional teams.
Engineer, Development III
Core Team Leader in engineering overseeing complex product design and commercialization at Southwire. Leading multi-discipline teams to enhance product performance and ensure safety compliance.
Manager, Demand Engineer
Manager of Demand Engineering optimizing revenue funnels at BeyondTrust. Focused on AI-powered workflows, attribution models, and cross-functional collaboration.
L1 – Reports Engineer
Reports Engineer providing customer support and report customization for an international SaaS company. Working remotely with a focus on communication and problem-solving in client interactions.
Application and Promotion Engineer
Technical Sales Engineer focusing on electrification solutions in Southern Brazil. Developing client relationships and technical sales activities to drive electrification projects.
SCADA Engineer – Ignition Exp Required
SCADA Front-End Engineer developing intuitive user interfaces for mission critical data center applications using Ignition SCADA. Collaborating cross-functionally to improve user experience and data visualization.