Member of Technical Staff, Coding Research
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Coding research specialist evaluating frontier AI agents through benchmarks, datasets, and failure analysis. Building research tooling and methodologies with software engineers and AI researchers.
Responsibilities:
- Design and own evaluation frameworks for advanced coding agents
- Develop benchmark specifications, scoring methodologies, rubrics, and quality standards
- Establish rigorous methods for measuring coding-model performance across diverse software-engineering tasks
- Define objective criteria for correctness, reasoning quality, robustness, and task completion
- Maintain methodological rigour, reproducibility, and consistency across evaluation workflows
- Develop high-quality datasets, golden examples, and structured evaluation protocols
- Design technical tasks for reliable assessment of frontier coding systems
- Build data and evaluation workflows supporting model development and iterative improvement
- Identify benchmark or dataset coverage gaps and develop new evaluation categories
- Analyse coding-agent behaviour and identify systematic weaknesses, failure modes, and performance limitations
- Investigate incorrect reasoning, implementation errors, tool-use failures, and incomplete task execution
- Translate findings into recommendations for model training and evaluation
- Design experiments testing hypotheses about coding-model capabilities
- Build tooling and infrastructure for large-scale experimentation, data generation, review workflows, and evaluation pipelines
- Automate technical processes to improve evaluation efficiency and research velocity
- Collaborate with researchers, engineers, and applied AI teams
- Contribute to technical reports, benchmark studies, research documentation, and external-facing research initiatives
- Communicate complex technical findings to specialist and broader technical audiences
Requirements:
- Strong software-engineering background with expertise in Python, C++, or comparable programming languages
- Minimum of 3 years of experience in software engineering, machine learning, AI research, evaluation, or a related technical discipline
- Experience designing, reviewing, or validating technical assessments, benchmarks, coding tasks, or evaluation methodologies
- Familiarity with large language models, coding agents, reinforcement learning, model evaluation, or related AI systems
- Proven ability to build tooling, automate workflows, and improve technical processes through systematic experimentation
- Strong analytical skills and ability to investigate complex model behaviour and technical failure modes
- Excellent written and verbal communication skills
- Ability to operate effectively in fast-moving research environments with significant ambiguity and evolving priorities
- Experience with frontier AI systems, coding agents, or model-evaluation research is advantageous
- Experience designing benchmarks or datasets for machine-learning systems at scale is strongly valued
- Familiarity with agentic workflows, tool use, reinforcement learning, or post-training methodologies is beneficial
- Publications, open-source contributions, or demonstrated technical leadership in AI, machine learning, or software engineering are advantageous
- Work must be completed without using confidential or proprietary information belonging to any employer, client, institution, or other third party
Benefits:
- Fully remote work
- Full-time engagement
- Opportunity to contribute to frontier AI research and development
- Collaboration with researchers, engineers, and applied AI teams
- Opportunity to contribute to technical reports, benchmark studies, research documentation, and external-facing research initiatives
Similar Jobs
Member of Technical Staff, Frontier AI
Frontier AI technical professional owning research evaluation, ML-oriented data design, and failure analysis. Improving deployed model and agent performance through evidence-based iteration.
Member of Technical Staff, Enterprise AI
Enterprise AI researcher improving machine-learning evaluation, datasets, and agentic workflows for 24-MAG. Analyzing failure modes and experimental results across enterprise AI systems.
Software Engineer
Software Engineer developing full-stack applications and migrating data to AWS for First Advantage’s fulfillment team. Automating workflows, resolving technical issues, and mentoring engineers.
Landscape Developer
Landscape Developer advancing Commonland’s nonprofit landscape-restoration network across the UK and Ireland. Building partnerships, fundraising strategies, and nature-finance mechanisms for regenerative change.
Engineering Intern
Engineering Intern creating 2D and 3D CAD drawings for Custom Truck One Source’s specialized truck and equipment solutions. Maintaining revisions and resolving drawing errors with contractors.
z/OS System Programmer
z/OS System Programmer maintaining and modernizing mission-critical mainframe infrastructure for DXC Technology. Supporting z/OS operations, LPAR projects, security, performance, and distributed EU teams.
Senior Associate – Platform Development Engineering
Senior Associate developing scalable nCino lending and CRM platforms for Atlantic Union Bank. Defining architecture, roadmaps, requirements, and platform support.
Senior Associate – Platform Development Engineering
Senior Associate developing scalable nCino, Salesforce, and lending/CRM platforms for Atlantic Union Bank. Defining architecture, roadmaps, requirements, and technical solutions.
Protection & Control Engineering Associate II
Protection and Control Engineering Associate designing substation drawings and connection diagrams for Mesa Associates’ power delivery projects. Using AutoCAD for substation engineering design.
AI/ML Document Processing Developer
AI/ML developer building AWS Bedrock pipelines to classify and extract healthcare document data. Operating scalable Python backends, OCR, and structured-data workflows for The ACI Group.
Medior 3D XR Developer
Medior 3D XR Developer building Unity-based medical simulations for SyncVR Medical’s hospital XR platform. Creating AI-enhanced training modules with C#, xAPI, and Quest 3 optimization.
Senior Dynamics MPOS Developer
Senior D365 Commerce developer customizing MPOS and Store Commerce solutions for Pandora’s retail technology ecosystem. Supporting integrations, deployments, production issues, testing, and scalable releases.
Odoo Developer
Odoo Developer customizing Python, JavaScript, and PostgreSQL-based ERP applications. Building integrated Odoo solutions for global clients through a Philippine remote staffing firm.
Software Engineer
Software Engineer building React/TypeScript tools that help multilingual learners access rigorous classroom content. Extending Medley Learning’s Chrome extension, web apps, and AI-powered instructional supports.
Developer Research Coordinator – Contract
Developer Research Coordinator supporting MLH’s developer research through participant panels, qualitative analysis, and census studies. Fully remote contract role serving technology companies and developer communities.
C# Developer
C# Developer building scalable .NET backend services, APIs, and automation tools. Supporting FyerX’s internal infrastructure and diverse client technology platforms.
Senior Power Apps Developer – Power Platform, Microsoft 365
Senior Power Apps Developer building Power Platform applications, automation, and integrations for Booker DiMaio’s IRS Microsoft 365 contract. Supporting secure federal business solutions.
Software Engineer
Part-time Software Engineer evaluating AI-generated technical responses and documentation for 24-MAG’s remote consulting platform. Scoring accuracy, clarity, and usefulness with detailed written rationales.
Member of Technical Staff, Finance Research
Finance AI researcher developing benchmarks and evaluation frameworks for advanced financial reasoning and agentic systems. Conducting applied research across AI models, enterprise workflows, and quantitative finance.
Software Engineer
Contract Software Engineer contributing code, debugging, optimization, and documentation across major programming languages. Reviewing technical submissions to improve AI training data quality.