Clinical AI Evaluation Specialist
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Clinical AI Evaluation Specialist defining quality standards for AI outputs at CentralReach. Collaborating on evaluation frameworks and monitoring systems in behavioral healthcare using AI.
Responsibilities:
- Contribute to evaluation frameworks for clinical AI products, including defining acceptance criteria, test plans, clinically relevant rubrics, and performance benchmarks.
- Conduct structured sampling reviews of AI-generated outputs, assessing across criteria alignment with ABA principles and BACB ethical standards.
- Design and implement monitoring automation in collaboration with engineering, including automated quality checks, alerting thresholds, and drift detection for AI systems.
- Contribute to the creation and maintenance of reusable governance artifacts including system cards, monitoring playbooks, evaluation of SOP templates, and risk assessment documentation.
- Assist with pre-deployment validation testing for new AI products and features, executing defined testing and results of documentation.
- Participate in risk tiering assessments for new and evolving AI products, providing clinical and evaluative perspectives.
- Support incident investigation for AI-related issues if/as needed and in alignment with clinical best practice and CR Development Policy.
- Collaborate with product management and engineering teams to provide input during product planning and development, ensuring governance considerations are integrated early.
- Contribute to the development of evaluation methodology and standards for clinical AI use cases, documenting approaches and refining processes based on findings and emerging best practices.
- Contribute to AI literacy and training content for internal teams and customers, drawing on hands-on clinical evaluation experience.
- Stay current with developments in AI evaluation methodology, LLM behavior, ABA best practices, and relevant regulatory and ethical standards.
- Support CentralReach's thought leadership in responsible AI through contributions to conference presentations, professional publications, or applied research as opportunities arise.
Requirements:
- Board Certified Behavior Analyst (BCBA) in good standing
- 2+ years of demonstrated experience in quality assurance, evaluation, data analysis, or technical assessment roles, preferably within healthcare or behavioral health settings.
- Demonstrated experience with large language models, generative AI, and their applications and limitations in clinical or professional contexts.
- Experience with prompt/context engineering, AI output evaluation, or AI workflow design.
- Experience with statistical sampling methodologies or healthcare quality assurance frameworks.
- Familiarity with AI governance frameworks (NIST AI RMF, EU AI Act risk categories, AMA governance guidance).
- Experience developing evaluation rubrics, scoring frameworks, or quality measurement tools.
- Background in clinical informatics, research methodology, or systematic quality improvement.
- Demonstrated proficiency with data analysis tools and comfort working with quantitative and qualitative evaluation data.
- Strong analytical skills with the ability to develop systematic, evidence-based evaluation approaches.
- Familiarity with HIPAA compliance, healthcare data standards, and ethical considerations in AI.
- Excellent written and verbal communication skills with the ability to document methodology clearly and convey findings to clinical, technical, and leadership audiences.
- Demonstrated ability to work collaboratively across clinical, technical, and product teams.
- Experience in a remote work environment with strong self-direction and accountability.
Benefits:
- Health benefits
- Generous PTO
- 401(k) matching
- Paid parental leave
- Hybrid work schedules
- Career development support
- Wellness programs
Similar Jobs
IT, AI Development Intern
IT/AI Development Intern providing hands-on exposure to software engineering and AI technologies at MIRA Safety. Collaborate on projects related to AI-powered tools, e-commerce, and analytics dashboard development.
Senior Forward Deployed Engineer – AI Agent
AI Agent Engineer developing state-of-the-art AI Agents to solve practical customer problems. Leveraging AI and ML technologies for deploying efficient solutions in Cresta's platform.
AI Model Lead
AI Models Lead overseeing language model adaptation for EHS, Quality, and GRC workflows. Leading team development and technical standard setting in regulated contexts to ensure correct model performance.
AI Domain Lead
AI Domain Lead defining and maintaining standards for AI models in regulated industries at Ideagen. Focused on ensuring models are reliable and fit for real-world decision-making.
Math AI Training Expert
Math AI Training Expert designing advanced mathematical problems for AI model training and evaluation. Requires Master's or PhD in Mathematics, strong Python skills, and attention to detail.
Math AI Training Expert
Math AI Training Expert designing challenging mathematical problems for AI training data. Collaborating to improve model reasoning in technical domains using Python and advanced mathematical techniques.
Math AI Training Expert
Math AI Training Expert designing rigorous mathematical problems for AI training. Working remotely to create deterministic problems improving AI's reasoning capabilities.
AI Value Partner, Customer Analytics
AI Value Partner at Cresta analyzing data to measure customer impact and support success teams. Collaborating with stakeholders, designing experiments, and building analytics dashboards.
Contract Practitioner – AI Skills Assessment Validation
Conducting AI skills assessments remotely for validation purposes, focusing on various AI disciplines. Engaging in practitioner-level tasks without instructional design or content creation responsibilities.
Senior Data Engineer, AI Engineer
Senior Data & AI-Engineer developing data-driven and AI solutions for healthcare markets. Collaborating closely with clients and internal teams to integrate advanced technologies.
Senior AI and Data Protection Engineer
Senior AI and Data Protection Engineer leading technical delivery for data classification, labeling, and governance. Collaborating on data loss prevention and security for AI integrations.
AI Analyst
AI Analyst managing the lifecycle of AI solutions in a remote role. Ensuring technical practices and alignment with organizational goals.
Co-Founder, Tech Partner – AI, Open Source
Co-Founder / Tech Partner developing open-source-based AI infrastructure for data management. Building a modular system with a focus on sustainability and privacy in Estonia.
VP Enterprise AI Enablement
Vice President leading EnerSys' enterprise artificial intelligence strategy and execution model for global business value. Overseeing AI governance, investment prioritization, and cross-functional orchestration.
Senior Compiler Engineer – AI
Machine Learning Compiler Engineer at NVIDIA focusing on applied machine learning and compiler technologies. Innovating solutions within diverse technical fields and leading the development of global products.
Director of AI Strategy – Enablement
Director of AI Strategy & Enablement at Fullbay, driving AI initiatives across the organization. Collaborating with cross-functional teams to optimize efficiency through AI solutions.
Senior AI Community Engineer
Senior AI Community Engineer creating opportunities for developers in voice AI. Running events, fostering community, and measuring activation success in the Bay Area.
AI Growth Marketing Lead
AI Growth Marketing Lead responsible for multi-channel demand generation strategies. Driving engagement and revenue for Diagrid's developer-first SaaS offerings.
Forward Deployed Engineer, AI - Advertising
Forward Deployed Engineer ensuring technical success of AI implementations with a focus on tailored solutions. Working with clients to adapt AI solutions and maintain security and standards.
Forward Deployed Engineer, AI – Advertising
Forward Deployed Engineer responsible for critical AI implementations at Truelogic. Collaborating with various teams to deliver bespoke AI solutions for top clients in digital marketing.