AI Policy Generalist
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
AI Policy Generalist evaluating AI model behavior for Handshake’s human-data business. Applying customer policies to ambiguous conversations and improving evaluation frameworks.
Responsibilities:
- Learn customer policies, definitions, taxonomies, and evaluation rubrics
- Evaluate user requests and AI model responses within full conversation context
- Classify cases according to the most applicable policy category
- Distinguish closely related labels, severity levels, and policy boundaries
- Select defensible classifications for ambiguous cases
- Write concise, evidence-based rationales citing policy language and conversation details
- Identify policy gaps, contradictions, unclear definitions, and emerging edge cases
- Raise questions when guidance does not resolve a case
- Participate in calibration discussions with evaluators, project leads, policy teams, and researchers
- Apply customer policies consistently without substituting personal beliefs
- Maintain accuracy and attention to detail across repeated evaluations
- Incorporate feedback and apply clarified guidance
- Help improve evaluation frameworks, examples, decision rules, and quality standards
- Move between projects covering different policy domains and customer needs
Requirements:
- Must be based in the United States
- Must be authorized to work lawfully in the United States for Handshake
- Must be able to work Monday–Friday from 8AM–5PM PT as core hours
- Ability to engage carefully, responsibly, and sustainably with sensitive material, including sexual content, emotional distress, self-harm, suicide, violence, weapons, abuse, exploitation, and discrimination
- Sound judgment and consistent work quality
- Ability to learn unfamiliar subject matter quickly
- Ability to communicate clearly and precisely in writing
- Ability to apply detailed policies, definitions, taxonomies, rubrics, and evaluation frameworks
- Ability to distinguish closely related labels, severity levels, and policy boundaries
- Ability to write concise, evidence-based rationales
- Ability to participate in calibration discussions and incorporate feedback
- Prior AI evaluation experience is helpful but not required
- Strong candidates may come from quality assurance, research, editing, law, teaching, operations, trust and safety, content moderation, social science, policy, investigations, compliance, or customer support
Benefits:
- W-2 employment classification
- Monday–Friday schedule
- Core hours of 8AM–5PM PT
- Remote work arrangement
Similar Jobs
AI Solution Lead
AI enablement lead scaling responsible enterprise AI across LG Energy Solution Vertech. Evaluating solutions, building governance, training employees, and driving measurable adoption.
AI Tools Video Tutorial Creator
AI tutorial creator producing narrated walkthroughs for Study.com’s generative-AI college courses. Demonstrating practical LLM workflows for non-technical learners.
Data Privacy and Responsible AI Manager
Data privacy and responsible AI manager for Accuray’s cancer radiotherapy systems. Assessing privacy, vendor, AI, and regulatory risks across enterprise operations.
Sales Leader – Native-AI Practice
Enterprise sales leader driving net-new OpenAI consulting and implementation engagements for Blend, an AI services provider. Building North American pipeline, closing strategic accounts, and developing co-sell motions.
Gen AI Creative Technologist
Gen AI Creative Technologist building agentic workflows and creative systems for Pencil’s AI marketing platform. Enabling faster, scalable, on-brand advertising production.
Marketing Expert – Organic Growth, AI Agent Environments
Organic Growth expert designing and evaluating simulated SEO/AEO marketing tasks for Mercor. Creating realistic environments and rubrics to train and improve frontier AI models.
AI Session Annotator, French
French AI session annotator reviewing voice-and-camera assistant sessions for Mercor, which helps AI labs train frontier models. Scoring visual accuracy, language, timing, and safety warnings.
AI Safety Expert, English – Bengali
AI safety red teamer probing frontier AI models for Mercor. Testing adversarial vulnerabilities and producing human data, datasets, and actionable reports.
AI Session Annotator, English
AI session annotator evaluating English voice-and-camera assistant responses for Mercor’s AI training projects. Scoring visual accuracy, safety warnings, timing, and spatial-language use with evidence-based comments.
Marketing Expert – Lifecycle, AI Agent Environments
Lifecycle marketing expert designing realistic CRM and retention tasks for Mercor’s AI-agent training environments. Evaluating agent decisions, reference answers, and multi-tool workflows.
Generative AI Analyst, Traditional Chinese
Generative AI Analyst evaluating and annotating AI-generated content for Welo Data, a global AI data company. Reviewing multilingual data quality and providing structured feedback.
AI Systems, Zoho & Automation Specialist
AI automation specialist connecting Zoho, AI, CRM, WhatsApp, and data systems. Building reliable workflows for Panama real-estate brokerages.
AI Team Lead
AI Team Lead leading engineers and delivering automation, RPA, and LLM solutions. Driving AI strategy and adoption across Immense Group’s iGaming business.
Marketing Expert – Lifecycle, AI Agent Environments
Lifecycle marketing expert designing simulated environments to evaluate AI agents on real CRM and retention work. Creating briefs, rubrics, and reference answers for Mercor’s AI training projects.
Marketing Expert – Field Marketing, AI Agent Environments
Field marketing expert designing realistic task environments for Mercor’s AI training projects. Evaluating agent performance across events, pipeline recovery, sponsorships, and multi-tool workflows.
Senior Technical Director, AI/ML
AI/ML Senior Technical Director prioritizing cross-business AI initiatives for GE Electrification. Driving strategic roadmaps, risk management, and GenAI implementation with global teams.
Contract Trainer for Operations Management, Data & AI
Operations-Management-Honorartrainer:in für WBS TRAININGs digitale Weiterbildung. Coaching, Projektbewertung und praxisnaher Einsatz von Daten- und KI-Tools im Homeoffice.
Contract Trainer for Operations Management, Data & AI
Honorartrainer:in für Operations Management, Data und KI bei WBS TRAINING. Begleitung von Lernenden im Flipped-Classroom-Modell aus dem Homeoffice.
AI Enablement Engineer
AI Enablement Engineer building internal AI systems, agents, and integrations for Replicant's AI-powered customer-service platform. Enabling teams, managing AI foundations, and improving enterprise AI cost visibility.
Mid-Level Analyst/Developer, Java, Artificial Intelligence
Analista Java diagnosticando e documentando plataforma educacional interna com IA na Deliver IT. Mapeando requisitos, integrações e modernização sem desenvolvimento inicial.