AI Engineer, Quality – Evals
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
AI Engineer developing evaluation platforms and pipelines for AI agents at fintech startup. Collaborating across teams to ensure reliable AI performance and product quality.
Responsibilities:
- Design and build a unified evaluation platform that serves as the single source of truth for all of our agentic systems and audit workflows
- Build observability systems that surface agent behavior, trace execution, and failure modes in production, and feedback loops that turn production failures into first-class evaluation cases
- Own the evaluation infrastructure stack including integration with LangSmith and LangGraph.
- Translate customer problems into concrete agent behaviors and workflows
- Integrate and orchestrate LLMs, tools, retrieval systems, and logic into cohesive, reliable agent experiences
- Build automated pipelines that evaluate new models against all critical workflows within hours of release
- Design evaluation harnesses for our most complex Agentic systems and workflows
- Implement comparison frameworks that measure effectiveness, consistency, latency, and cost across model versions
- Design guardrails and monitoring systems that catch quality regressions before they reach customers
- Use AI as core leverage in how you design, build, test, and iterate
- Prototype quickly to resolve uncertainty, then harden systems for enterprise-grade reliability
- Build evaluations, feedback mechanisms, and guardrails so agents improve over time
- Work with SMEs and ML Engineers to create evaluation datasets by curating production traces.
- Design prompts, retrieval pipelines, and agent orchestration systems that perform reliably at scale
- Define and document evaluation standards, best practices, and processes for the engineering organization
- Advocate for evaluation-driven development and make it easy for the team to write and run evals
- Partner with product and ML engineers to integrate evaluation requirements into agent development from day one
- Take full ownership of large product areas rather than executing on narrow tasks
Requirements:
- Multiple years of experience shipping production software in complex, real-world systems
- Experience with TypeScript, React, Python, and Postgres
- Built and deployed LLM-powered features serving production traffic
- Implemented evaluation frameworks for model outputs and agent behaviors
- Designed observability or tracing infrastructure for AI/ML systems
- Worked with vector databases, embedding models, and RAG architectures
- Experience with evaluation platforms (LangSmith, Langfuse, or similar)
- Comfort operating in ambiguity and taking responsibility for outcomes
- Deep empathy for professional-grade, mission-critical software (experience with audit and accounting workflows are not required)
Benefits:
- Competitive compensation packages with meaningful ownership
- Flexible PTO
- 401k
- Wellness benefits, including a bundle of free therapy sessions
- Technology & Work from Home reimbursement
- Flexible work schedules
Similar Jobs
AI Engineer – Apprenticeship Program, Accelerated Track
Apprentice software engineer contributing to AI and Data Science projects at Flatiron School. Learning through a structured bootcamp and employer-partner mentorship.
Senior Applied AI Engineer
Sr. Applied AI Engineer at Zapier focusing on AI Platform capabilities and improving LLM/ML Ops systems. Collaborating across teams to build robust infrastructure for machine learning and generative AI.
Senior Fullstack AI Engineer – React Native, TypeScript
Senior Full-Stack AI Mobile Engineer at Devlane designing and shipping mobile products. Developing scalable backends and cross-platform UIs with a focus on generative AI tools.
Senior AI Architect
Senior AI Architect leading the design and delivery of production AI systems for enterprise clients. Working in a hands-on, client-facing role anchored in the Microsoft stack and extending across OpenAI and Anthropic platforms.
Senior Staff AI Engineer, Context Engineering
Sr. Staff AI Engineer at WEX designing high-performance software foundation for enterprise. Focus on distributed systems, cloud-native architecture, and next-generation AI workflows.
Senior Software Engineer – Context Engineering
AI Platform Engineer building high-performance software foundations for enterprise AI at WEX. Focused on distributed systems and cloud-native architecture in a remote role.
Mid-Level Software AI Engineer, Context Engineering
AI Platform Engineer developing and maintaining core software components for AI infrastructure at WEX. Focused on operational excellence and CI/CD automation in cloud-native environments.
AI Engineer
AI Engineer position at MDS focused on building AI agent systems using Claude and automating workflows. Collaborating with tech leads to enhance existing automation processes.
Senior AI Engineer, Agentic Systems
Senior AI Engineer at AI Technology Partners designing agentic systems solutions for enterprises. Leading architecture and integration efforts for robust, secure AI deployments while collaborating with enterprise clients.
Senior Applied AI Engineer
Senior Applied AI Engineer at Acuity Analytics responsible for building data pipelines and ML solutions. Collaborating closely with cross-functional teams to improve operational efficiency and delivery outcomes.
Junior AI Developer
Junior AI Developer at BenGen responsible for driving innovation in AI-centric projects. Collaborating with teams to build machine learning models and solve real-world problems.
Senior Product Manager – AI Platform
Senior Product Manager overseeing AI platform development at Cority. Leading cross-functional teams and managing product operations with a focus on AI solutions.
Founding AI Engineer
Founding AI Engineer at Paradox Machines, building AI applications and data infrastructure for a data-driven company. Collaborating with a senior team and solving real-world problems with AI.
AI Engineer Trainee
Apprenticeship opportunity for career switchers to learn software engineering skills. Involves bootcamp programs and hands-on projects with mentorship in a fully remote setup.
AI Engineer II
AI Engineer contributing to AI capabilities for healthcare data systems. Collaborating on projects from concept to deployment with cross-functional teams and ensuring high-quality deliverables.
AI Engineer
Agentic AI Engineer developing advanced AI agents focused on cybersecurity applications for national security. Join a cutting-edge team at Whitespace using machine learning solutions.
Full Stack AI Engineer
Full-Stack AI Engineer building AI agents and products for ANNA's innovative autism care platform. Collaborating with teams to enhance clinical workflows and operational efficiency.
Senior Staff AI Engineer, Architect
AI Architect responsible for design and evolution of AI systems at TELUS Digital. Integrating Generative AI into scalable and production-ready solutions.
AI Engineer
AI Engineer leveraging AI models to improve customer communication and market strategies at Teleport. Analyzing data and creating workflows within the product marketing team.
Senior AI Engineer – Enterprise GenAI, SAP Environment
Senior AI Engineer designing and scaling enterprise-grade AI and Generative AI solutions for PulseRise Technologies. Collaborating with teams to build intelligent agents to streamline core processes.