Director, AI Automation Engineering

Posted 15hrs ago

Employment Information

Industry

Education

Salary

Experience

Job Type

Location

Report this job

Job expired or something wrong with this job?

Job Description

Director building automated testing, evaluation, and release gates for Impiricus’s AI-powered healthcare commercialization platform. Ensuring safe, reliable agent and LLM deployments at scale.

Responsibilities:

Own infrastructure that continuously verifies whether autonomous agents take intended actions before and after production
Stage, test, and lifecycle-manage bots and agent workflows
Establish baseline quality metrics and coverage standards for AI initiatives
Build infrastructure to run automated research chains and agent evaluations at scale
Create automated evaluation frameworks for LLM outputs covering accuracy, safety, bias, and regression
Implement RAG evaluation, prompt regression testing, and golden-set benchmarking
Define and enforce quality thresholds before release
Partner with AI Researchers and Engineers on evaluation design for new model and agent patterns
Build release-gating infrastructure, including staged rollouts and rollback authority
Develop AI automation solutions to maintain software quality across the product portfolio
Integrate testing into GitHub Actions or equivalent for AI model and agent releases
Maintain test suites as models, prompts, and tools change
Collaborate with AI Solutions Architects, AI Engineers, Principal AI Product, and product teams on evaluation, launch readiness, QA, and feature validation
Document standards, runbooks, and patterns so non-engineers building in Sage can self-serve basic testing
Set technical standards and grow into leading a small automation team

Requirements:

5+ years building, evaluating, or governing production AI/ML systems, with direct experience in agentic AI, LLM-based automation, or autonomous system safety
Direct experience evaluating LLM outputs, including RAG evaluation, prompt regression, safety, and accuracy benchmarking
Experience building CI/CD pipelines with integrated test gates using GitHub Actions or similar
Comfortable working across AI platform infrastructure and product-facing AI features
Ability to operate independently as a senior IC and communicate quality standards to engineers and non-engineers
Experience with agent testing, bot lifecycle management, or AI agent orchestration platforms is a strong plus
Familiarity with LLM observability and evaluation tooling such as LangSmith, Braintrust, or custom harnesses is a strong plus
Background in healthcare, pharma, or regulated environments is a strong plus
Prior experience establishing a QA or automation function from scratch, with appetite to grow into leading a team
Proficiency in Python and standard testing frameworks such as pytest or equivalent

Benefits:

Medical, dental, and vision coverage for you and your dependents
On-demand healthcare concierge
Pre-tax HSA, FSA, and DCFSA savings options
Monthly employer HSA contributions if enrolled in a high-deductible plan
100% paid short- and long-term disability insurance
Life and AD&D insurance
Flexible vacation policy
Paid parental leave after 6 months
Remote work option
401(k) with company match

Director, AI Automation Engineering

Employment Information

Report this job

Job Description

Responsibilities:

Requirements:

Benefits:

Impiricus

Report this job

Similar Jobs

Element Fleet Management

Foresite Cybersecurity

munerio consulting GmbH

Cyber Tools and Solutions

Integres, LLC

EX Squared

Wabtec Corporation

Satellite Office

Koniag Government Services

UpDoc

Ford Motor Company

Rhythm Energy

Hamilton Company

Offshore 24/7

Evri

Dijital Team

Nimble Gravity

Upgrade, Inc.

Körber

Sequoia Connect