AI Evaluation Guidelines & Rubric Specialist
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
AI evaluation specialist developing precise rater guidelines and scoring rubrics for generative AI programmes. Translating complex requirements across finance, legal, retail, insurance, and sports domains.
Responsibilities:
- Translate programme requirements into clear, structured, and actionable instructions for human evaluators
- Develop guidelines applicable to standard scenarios and complex edge cases
- Define terminology, rating criteria, decision rules, exceptions, and escalation pathways
- Ensure instructions are accessible to raters while preserving domain-specific precision
- Design detailed scoring rubrics for evaluating generative AI outputs
- Establish measurable criteria for correctness, relevance, reasoning quality, completeness, and instruction adherence
- Create examples and counterexamples illustrating different performance levels
- Align evaluation frameworks with programme objectives and quality standards
- Review draft specifications for ambiguity, contradiction, missing information, and inconsistent terminology
- Identify instructions that may lead to conflicting interpretations across raters
- Revise guideline sets for reliable application with minimal escalation
- Document before-and-after improvements to written requirements and evaluation instructions
- Convert specialist-domain specifications into rater-ready guidance across finance, retail, insurance, legal, sports, and other domains
- Collaborate with subject matter experts on domain-specific terminology and professional judgment
- Preserve technical nuance while making instructions clear to non-specialist evaluators
- Maintain consistent structure and quality across domain-specific guideline sets
- Participate in onboarding, calibration, documentation review, and ongoing guideline refinement
Requirements:
- At least 3 years of professional experience in linguistics, instructional design, technical writing, content design, or a closely related field
- Direct experience developing or refining guidelines and rubrics for human evaluators in generative AI, RLHF, or model-assessment programmes
- Direct experience developing rater guidelines or rubrics for generative AI or RLHF programmes is required
- Demonstrated ability to resolve ambiguity and contradiction in complex written specifications
- Experience translating specialist requirements into clear and practical instructions
- Ability to work effectively across multiple subject-matter domains
- A portfolio or concrete examples showing measurable improvements to guidelines, rubrics, or instructional materials
- Demonstrable professional growth and increasing responsibility
- Reliable availability for at least 35 hours per week during weekdays
- Equivalent professional experience in AI evaluation, technical documentation, or guideline development may also be considered
- Applicants should be prepared to provide concrete examples of guideline or specification improvements
- Immediate availability is preferred
- Experience supporting large language model evaluation, reinforcement learning from human feedback, or AI training-data programmes
- Familiarity with annotation platforms, human-feedback workflows, and rater calibration processes
- Experience developing domain-specific guidance for finance, insurance, retail, legal, sports, or comparable fields
- Knowledge of controlled language, information architecture, taxonomy design, or content governance
- Experience conducting guideline usability tests or analysing inter-rater consistency
- Familiarity with version control, documentation systems, and structured authoring tools
- Previous collaboration with researchers, programme managers, engineers, and subject matter experts
Benefits:
- Full-time remote engagement
- Competitive hourly compensation of $40–$60 per hour depending on expertise and project scope
- W-2 contingent employment arrangement
- Remote work from anywhere in the United States
- Expected commitment of at least 35 hours per week during weekdays
- Onboarding and calibration opportunities
- Project scope and duration may be adjusted according to programme requirements and performance
Similar Jobs
Senior Marketing Strategy, AI Evaluation Specialist
Senior marketing strategist designing evaluation tasks and rubrics for generative AI models. Assessing AI-generated marketing content using brand, growth, and performance expertise.
U.S. Licensed Dermatologist – AI Dermatology Research
U.S.-licensed dermatologist reviewing skin images and AI-generated assessments for 24-MAG LLC’s dermatology research. Establishing clinical guidelines and improving medically grounded AI systems.
AI Safety Red Teamer
AI Safety Red Teamer identifying vulnerabilities in frontier AI systems for Mercor. Designing adversarial prompts and evaluating high-risk model behavior with AI researchers.
AI Safety Red Teamer
AI Safety Red Teamer adversarially testing frontier AI models for Mercor, which partners with AI labs to train and enhance them. Documenting vulnerabilities across high-risk domains.
AI Safety Red Teamer
AI Safety Red Teamer stress-testing frontier AI models for Mercor. Designing adversarial prompts and documenting vulnerabilities to improve AI alignment and safety.
AI Data & Insights Officer
AI Data & Insights Officer turning event and luxury-travel business data into reliable intelligence. Building RAG, dashboards and agent-performance insight for Worldwide Events Group.
AI Data & Insights Officer
AI Data & Insights Officer transforming Worldwide Events Group’s MICE and luxury travel data into reliable intelligence. Building RAG knowledge systems, dashboards and measurable agent performance insights.
Data & AI Analyst
Data & AI Analyst transforming insurance marketing data into actionable insights, dashboards, and AI-powered automation. Building scalable solutions with SQL, BigQuery, Tableau, Python, and emerging AI tools.
AI Safety Red Teamer
AI Safety Red Teamer stress-testing frontier AI models for Mercor. Designing adversarial prompts and documenting vulnerabilities across high-risk domains.
AI Safety Expert, English, Finnish
AI red team expert testing Mercor’s frontier AI models through jailbreaks, prompt injections, and adversarial data generation. Producing reproducible vulnerability reports that improve AI safety.
AI Director
AI Director leading production AI architecture, governance, and client strategy at Futurice, a data-and-AI consultancy. Growing UK capability through consulting, business development, mentoring, and hiring.
Independent Zen Ambassador – AI Solutions
Ambassadeur freelance prospectant des dirigeants et les connectant à des solutions IA et experts B2B. David assure la vente, avec rémunération au succès et bonus.
Junior AI, Product Enablement Specialist
Junior enablement specialist building AI-assisted cloud-security training labs and technical content for Sysdig. Supporting DevOps training delivery, workshops, automation, and courseware development.
Head of AI – Processes
Head of AI & Processes leading AI consulting, process transformation, and automation growth. Developing LM IT Services’ practice, team, portfolio, and international customer projects.
Delivery Director – AI Foundry, Anthropic Business
Delivery Director leading NewRocket’s enterprise AI and Anthropic services portfolio. Driving delivery excellence, client growth, responsible AI adoption, and operational performance.
AI Safety Red Teamer
AI Safety Red Teamer identifying vulnerabilities in frontier AI models for Mercor. Designing adversarial prompts and evaluating high-risk AI behaviors.
AI Safety Red Teamer
AI Safety Red Teamer stress-testing frontier AI models for Mercor. Identifying jailbreaks, hallucinations, and policy failures across high-risk domains.
AI Safety Red Teamer
AI Safety Red Teamer stress-testing frontier AI models for Mercor’s AI training projects. Identifying vulnerabilities and improving model alignment, robustness, and safety.
AI Safety Red Teamer
AI Safety Red Teamer stress-testing frontier AI models for Mercor. Identifying jailbreaks, vulnerabilities, and safety failures through adversarial evaluation.
AI Safety Red Teamer
AI Safety Red Teamer stress-testing frontier AI models for Mercor, which partners with AI labs to train models. Finding vulnerabilities and improving model safety through adversarial evaluation.