AI Quality & Evaluation Lead
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
AI evaluation lead building quality rubrics and validating judges for Samsung Health’s generated coaching. Samsung Food connects food, health, and home through personalized digital experiences.
Responsibilities:
- Build the experience-quality evaluation layer for AI-generated coaching in Samsung Health
- Author a failure taxonomy for coaching outputs
- Hand-grade at least 150 traces with open-coded notes, including at least 40 sparse-data synthetic profiles
- Categorize failures into 5–10 categories and quantify their frequency
- Freeze at least 50 traces as a holdout set
- Create binary pass/fail criteria for each taxonomy category
- Write annotation guidelines with pass and fail examples
- Double-code at least 30 traces, resolve disagreements, and maintain a guideline revision log
- Create judge prompts for every criterion
- Validate the LLM judge using true-positive and true-negative rates on development and holdout sets
- Run weekly readouts on helpfulness, relevance, and tone
- Document a revalidation routine triggered by model or prompt changes and quarterly regardless
- Produce a complete playbook covering grading, taxonomy updates, rubric revisions, judge revalidation, and weekly readouts
- Conduct a handover test enabling the internal owner to rerun validation and a weekly readout independently
- Obtain Head of Product sign-off on the taxonomy and rubric
- Collaborate with and hand over the method to an internal owner
Requirements:
- Experience running the full evaluation loop at least once on conversational or generated-text output
- Experience with error analysis on real traces
- Experience building a failure taxonomy
- Experience creating binary criteria and annotation guidelines
- Experience validating an LLM judge against personal labels
- Background in conversation design, AI product quality, model behaviour/policy, content design, human data operations, UX research with strong qualitative coding, RLHF, applied linguistics, or product management
- Ability to identify previously unnamed failure modes by reading output
- Ability to act as arbiter and document overrulings and rationale
- Understanding that rubrics are discovered through grading
- Ability to explain limitations of judge agreement rates
- Ability to write unambiguous annotation guidelines
- Comfort working in notebooks and spreadsheets
- No production coding required
- Nutrition, weight management, or behaviour change domain expertise is not required
- Must be comfortable being responsible for own taxes and having no paid time off
Benefits:
- Remote-first work arrangement
- Independent contractor setup
- Flexible remote work
- Global team of more than 100 people in over 30 countries
Similar Jobs
Senior Manager, AI Business Modernization
AI modernization leader advancing Hershey’s U.S. snacks business through AI products, agentic solutions, and measurable productivity gains. Driving adoption, governance, and value realization across functions.
Engineer III, AI SDLC Engineer
AI SDLC Engineer building agentic workflows for CrowdStrike’s cybersecurity platform. Improving developer productivity through LLM infrastructure, observability, model routing, and benchmarks.
Working Student, Sales Operations & AI
Working student supporting PTV Logistics, a transportation logistics software company, with AI-driven sales operations. Building analytics, improving CRM data quality, and optimizing business processes.
Manager – Transformation & Process Design, AI, Agentic Organization
Transformationsmanager gestaltet KI-Agentenorganisation und KI-gestützte Prozesse bei Meliorate, einer Managementberatung für asset-intensive Unternehmen. Aufbau von Governance, Workflows und Veränderungskompetenzen.
AI Automation Expert – Microsoft M365, Copilot
AI Automation Expert building Microsoft 365, Copilot Studio, and Power Automate solutions. Delivering workshops and practical AI automation for mid-sized businesses.
Co-Founder – Tech Partner, AI, Open Source
Technical co-founder building open-source AI and data infrastructure in Estonia. Designing platform architecture, AI integrations, and ERP adapters while shaping product strategy and startup partnerships.
Generative AI Artist
Generative AI Artist creating AI-driven visuals, sequences, and previs for Rodeo FX’s visual-effects and advertising productions. Building ComfyUI workflows and collaborating with compositing, 3D, and art-direction teams.
Account-Based Marketing Strategist – Train AI
ABM Strategist shaping account-specific marketing for RWS TrainAI’s enterprise AI data services. Driving personalized campaigns, sales enablement, and measurable account growth.
Applied AI Sales Executive
Applied AI Sales Executive selling Cloudera-supported Mistral enterprise AI solutions. Owning complex enterprise deals from proof of concept through expansion across Cloudera data estates.
AI and Innovation Intern
Estagiário(a) explorando agentes de IA generativa, LLMs e dados na Zup Innovation. Testando soluções para otimizar processos e criar produtos digitais.
Analyst – Commercial Contract Administrator, Analytics, Automation & AI Innovation
Commercial contract administrator processing pharmaceutical rebates and detecting revenue leakage. Using SQL, Excel, Power BI, and AI automation to improve Lilly’s payment operations.
AI Transformation Office Lead
JLL AI transformation leader establishing a global office to implement AI across real estate services. Delivering enterprise-scale savings through platforming and operational optimisation.
AI Automations Specialist
AI automation specialist building agents, workflows, and outreach systems for a personal financial advisor. Managing award travel, customer calls, prospecting, and executive support remotely.
Enterprise AI Administrator – GenAI, Copilot, SaaS Platforms
Ford AI Platform Administrator managing enterprise Copilot, Gemini, and Anthropic platforms. Securing AI governance, agent lifecycles, compliance, usage, and operational reliability.
Head of AI Expansion
Head of AI Expansion driving global adoption of Tether’s QVAC and decentralized AI infrastructure. Building enterprise, government, and technology partnerships for privacy-preserving AI.
Head of AI Expansion
Head of AI Expansion driving global adoption of Tether’s QVAC and decentralized AI ecosystem. Building strategic partnerships, go-to-market execution, and a worldwide AI expansion team.
AI Safety Red Teamer
AI Safety Red Teamer stress-testing frontier AI models for Mercor. Designing adversarial prompts and documenting vulnerabilities to improve model safety and alignment.
AI Automation Specialist
AI and Automation Specialist building integrations, workflows, and AI tools for Rex Moore’s construction and enterprise operations. Automating processes, activating data, and supporting organization-wide adoption.
Instructor – Agentic AI
Part-time online Agentic AI instructor delivering hands-on training for Simplilearn bootcamp learners. Teaching AI agents, LLMs, RAG, machine learning, and automation through live classes and mentorship.
Director, Cloud & AI Infrastructure Operations
Ford Credit director leading secure GCP, Kubernetes, and AI infrastructure operations. Driving resilience, security, automation, cost efficiency, and technical team development.