AI Agent Evaluator
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Evaluating Hippocratic AI’s clinical agents for symptom triage, diabetes medication titration, and travel health guidance. Providing safety-focused feedback to improve AI-driven healthcare tools.
Responsibilities:
- Evaluate recorded and simulated AI agent interactions involving acute symptom triage and assess escalation and de-escalation decisions
- Review agent conversations related to insulin and diabetes medication titration for clinical accuracy and guideline adherence
- Evaluate agent-delivered travel health guidance, including pre-travel counseling, vaccination recommendations, and destination-specific risk assessment
- Identify clinical safety concerns, edge cases, and gaps in triage logic or clinical reasoning
- Score and annotate agent performance using structured evaluation rubrics
- Provide actionable written feedback to clinical and product teams to inform agent refinement
- Participate in case review and calibration sessions with other clinical SMEs
- Help develop and refine test scenarios for ongoing agent quality assurance
Requirements:
- Registered Nurses: active, unrestricted RN License
- Nurse Practitioners: active, unrestricted Nurse Practitioner license in the United States, with current board certification (ANCC or AANP)
- Minimum of 5+ years of direct clinical experience in primary care, urgent care, telephone/virtual triage, or a comparable acute-care setting
- Clinical experience managing patients with diabetes, including insulin titration
- Strong, current working knowledge of relevant triage protocols and clinical practice guidelines
- Ability to critically evaluate clinical reasoning and identify inaccuracies or safety risks in written or recorded clinical interactions
- Comfortable working independently in a remote, asynchronous contract role
- Strong written communication skills for delivering structured clinical feedback
- Experience in telephone triage or nurse advice line settings (recommended)
- Experience in travel medicine, occupational health, or pre-travel consultation (recommended)
- Familiarity with CDC and other travel medicine guidelines (recommended)
- Prior experience in clinical quality assurance, utilization review, or case review (recommended)
- Familiarity with AI-assisted clinical tools or digital health platforms (recommended)
Benefits:
- Fully remote work
- Flexible scheduling
- Hourly compensation commensurate with experience
Similar Jobs
Sustainability Metrics Governance, AI Specialist
Sustainability metrics specialist improving ESG data governance and AI-enabled reporting at GE Vernova. Monitoring KPIs, validating disclosures, and deploying scalable automation solutions.
Generative AI Analyst, Portuguese
Generative AI Analyst reviewing Portuguese AI content for Welo Data’s global AI training systems. Annotating multimedia data and performing quality checks to improve advanced AI datasets.
AI User Interview Specialist
Paid 60-minute Terac interview examining how professionals use generative and analytical AI tools. Sharing workflow pain points and proposing future capabilities for AI products.
Director, AI and Technology Enablement
Director leading Salesforce, AI workflow, and technology enablement for TMG, a leadership advisory firm. Translating business priorities into practical systems, data processes, vendor implementations, and user adoption.
Internship – AI-Based Semantic Mapping for Autonomous Driving
CARIAD automotive software intern researching radar-camera fusion and semantic mapping for autonomous driving. Developing and evaluating multimodal perception methods for complex vehicle environments.
AI Subject Matter Expert
AI subject matter expert advising Veterans Affairs on responsible AI governance, risk, and lifecycle management. Translating AI strategy into operational guidance for healthcare programs and leaders.
AI Consultant – Contract
AI Consultant leading enterprise AI initiatives focused on evaluating, designing, and implementing AI solutions. Collaborating with stakeholders and ensuring compliance with governance and security standards.
Director, Safety and PV Systems – Innovation, AI
Director of Safety and PV Systems at Syneos Health responsible for technology solutions and cross-functional execution. Leading AI innovations and managing senior teams for patient safety analytics.
AI Solution Strategist
AI Solution Strategist at NICE developing conversational AI agents. Collaborating with cross-functional teams to define, design, and launch AI solutions for customer success.
Associate Director, AI for Manufacturing and Operations
AI Subject Matter Expert at RTX developing scalable AI solutions for manufacturing and operations. Collaborating with teams to enhance productivity, quality, and decision-making in factories.
AI Operations & Automation Intern – Summer 2026
AI Operations & Automation Intern focusing on enhancing VSC workflows and automating processes. Engage with teams to identify efficiencies and implement AI solutions during a summer internship.
AI Training Specialist, Data Annotation
AI Training Specialist responsible for data annotation to train AI systems and improve accuracy. This fully remote role focuses on data labeling and quality assurance.
Founding Director – AI Digital Strategy
Founding Director leading AI-driven digital strategy engagements for mid-market clients. Overseeing migrations, strategy layers, and AI agent deployments in a pioneering digital landscape.
AI Technology Risk Manager
AI Technology Risk Manager managing technology risks associated with machine learning and generative AI solutions. Partnering with teams to ensure ethical and compliant AI practices at Empower.
Commercial Real Estate Specialist – AI Training Project
Consultancy role seeking experienced Commercial Real Estate Specialist to analyze property scenarios. Working remotely with a focus on market analysis and feasibility studies.
Senior Director, AI, Architecture – Insurance Industry Experience Required
Senior Director leading AI and architecture within The Baldwin Group, focusing on insurance sector innovations. Driving enterprise technology and AI capabilities for scalable growth and operational excellence.
Working Student Logistics, AI
Supports AI-driven logistics software development by testing and modeling operational processes. Collaborates with product and engineering teams in a remote-first culture.
AI Creative Designer
AI Creative Designer leveraging AI tools for visual storytelling and creative collaboration at an innovative marketing agency. Design and create motion graphics, prototypes, and campaign assets.
AI Creative Designer
AI Creative Designer leveraging emerging technology to develop visual concepts and creative outputs. Collaborating with teams to create high-quality visual work for an influential brand.
AI Users and Non-AI Users Interview on Privacy and Assistant Usage
Participate in a paid study discussing AI assistant usage and privacy perspectives. Engage in a 60-minute remote video interview with Terac.