AI Quality & Evaluation Lead
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
AI Quality & Evaluation lead enabling scalable AI adoption through quality standards and evaluation frameworks for various teams. Ensure accuracy and compliance across AI solutions aligned with trust expectations.
Responsibilities:
- Enable responsible and scalable AI adoption by defining technical quality standards, evaluation framework and control requirements across the AI lifecycle.
- Partner with AI Engineering, Data Scientists, QA CoE, and Product teams to define quality and performance requirements.
- Define quality-by-design partnerships, embedding quality considerations into model and system architecture.
- Design and maintain structured evaluation frameworks that assess AI systems against quality standards.
- Develop automated metrics for Generative AI performance including Groundedness and Faithfulness.
- Provide objective, data-driven evaluation outputs that support AI governance reviews.
Requirements:
- 5–8+ years of experience in Data Science, ML Engineering, or AI Quality, with a focus on evaluation and statistical validation.
- Practical experience partnering with engineers to design RAG, LLM-based Agents, or traditional ML pipelines.
- Expert-level Python (Pandas, Scikit-learn) and experience with evaluation frameworks (e.g., RAGAS, TruLens, or MLflow).
- Demonstrated ability to translate abstract trust concepts into mathematical metrics and enforceable technical controls.
- Demonstrated ability to work without direct authority, driving quality adoption across engineering and product teams through enablement.
- Ability to bridge the gap between high-level governance policy and low-level code implementation.
- Bachelor’s degree in a technical field (e.g., Computer Science, Computer Systems Design) or equivalent professional experience.
Benefits:
- Health, dental and vision coverages starting Day One.
- Wellbeing programs.
- Retirement plans with contribution matching.
- Generous time off.
- Parental leave.
- Continuing education and career growth opportunities.
- Flexible working arrangements.
Similar Jobs
Senior AI Community Engineer
Senior AI Community Engineer creating opportunities for developers in voice AI. Running events, fostering community, and measuring activation success in the Bay Area.
AI Growth Marketing Lead
AI Growth Marketing Lead responsible for multi-channel demand generation strategies. Driving engagement and revenue for Diagrid's developer-first SaaS offerings.
Forward Deployed Engineer, AI - Advertising
Forward Deployed Engineer ensuring technical success of AI implementations with a focus on tailored solutions. Working with clients to adapt AI solutions and maintain security and standards.
Forward Deployed Engineer, AI – Advertising
Forward Deployed Engineer responsible for critical AI implementations at Truelogic. Collaborating with various teams to deliver bespoke AI solutions for top clients in digital marketing.
Forward Deployed Engineer – AI, Advertising
Responsible for deploying and adapting AI solutions for clients while ensuring adherence to standards. Collaborate with Product Managers to analyze technical needs and improve AI implementations.
Forward Deployed Engineer, AI – Advertising
Forward Deployed Engineer (AI) responsible for deployment and adaptation of AI implementations at a global marketing network. Collaborates with teams and ensures technical success for clients.
AI-Enabled Operations & Business Support VA
Virtual Assistant providing business support in operations, client coordination, and marketing execution for a Construction Technology company.
Clinical Practice Nurse Educator – AI Response Evaluation
Clinical Practice Nurse Educator at Elsevier evaluating AI-generated nursing educational content. Remote part-time role supporting nursing education with a focus on accuracy and professional assessment.
Contract Practitioner – AI Foundations
Practitioner role with Nectar Inc focused on AI Foundations skills assessment validation. Engage in a short-term contract to complete assessments and surveys remotely.
AI Automation Specialist
AI Automation Specialist working with a digital marketing agency to design and optimize automation systems. Focused on AI and internal processes while enhancing client services.
AI Product Sales Specialist
AI Product Sales Specialist at Snowflake driving sales for AI/ML solutions in the enterprise market. Collaborating with teams to ensure successful customer adoption and engagement.
AI Enablement & Governance Lead – Policy, Controls & Compliance
AI Enablement & Governance Lead managing AI governance policies, ensuring compliance and risk management at Alight. Partnering with teams to enable responsible AI adoption and scalability.
Senior Marketing Manager – AI
Senior Marketing Manager defining and operationalizing AI-driven marketing strategy at Delinea. Collaborating across teams to implement effective AI marketing initiatives.
AI Instructional Lead – 1099 Contract, Part Time
AI Instructional Lead at CDW developing innovative learning solutions in Artificial Intelligence. Collaborating with teams to enhance educational approaches and support client needs in AI solutions.
Associate Director, AI & Technology Learning Enablement
Associate Director driving learning enablement for AI and technology at IQVIA. Leading skills initiatives to enhance workforce capabilities and productivity.
AI Solutions Lead
AI Solutions Lead focusing on artificial intelligence establishment and implementations across the agency. Collaborating with leadership to identify opportunities that drive operational efficiencies.
AI Specialist, Claude Code, VibeCoding
AI Specialist developing automated workflows with Claude Code for easycosmetic's e-commerce platform. Collaborating with teams to enhance process optimization and AI-driven solutions in a modern work environment.
AI Automation Specialist, Enablement
AI Specialist responsible for automating processes and developing internal AI capabilities for INTERSPORT Digital GmbH. Involves strategy management, engineering, and governance in automation.
Junior AI Agent Builder
AI Engineer helping teams adopt AI systems through building and automating workflows using cutting edge tools. Fully remote position based out of Buenos Aires, Argentina.
Junior AI Agent Builder
AI Engineer building automation workflows and AI tools for Adaptive Teams. Fully remote position focused on practical AI applications and team collaboration.