AI Evaluation Specialist
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
AI Evaluation Specialist assessing AI-generated outputs, reasoning, and tool use for 24-MAG LLC’s remote consulting platform. Providing rubric-based quality evaluations and actionable written feedback.
Responsibilities:
- Evaluate AI-generated outputs against detailed rubrics, guidelines, and defined quality standards
- Assess responses for accuracy, relevance, completeness, reasoning quality, and adherence to instructions
- Apply consistent and impartial judgement across large volumes of evaluation examples
- Identify outputs that fail to satisfy important quality or task requirements
- Maintain reliable assessment standards across repeated evaluation workflows
- Identify reasoning gaps, logic errors, inconsistencies, unsupported conclusions, and tool-use failures
- Analyse where AI-generated outputs diverge from expected reasoning or quality standards
- Document recurring model weaknesses and opportunities for improvement
- Produce clear, concise, and actionable written feedback on strengths and areas for improvement
- Explain the reasoning behind evaluation decisions and quality scores
- Maintain detailed, transparent, traceable, and reproducible assessment documentation
- Participate in rubric interpretation and ambiguous-case discussions
- Help refine assessment criteria as AI models and project requirements develop
- Contribute insights supporting process optimisation and evaluation best practices
- Collaborate with other reviewers to improve alignment and reliability across evaluation workflows
Requirements:
- Experience in grading, quality assurance, editorial review, assessment, annotation, or another field requiring careful analysis and detailed feedback
- Advanced, regular use of AI assistants such as ChatGPT, Claude, or comparable tools for professional work and productivity
- Strong ability to synthesise complex information and communicate conclusions clearly in writing
- Experience with process improvement, rubric development, operational quality assessment, or structured evaluation workflows is advantageous
- Strong critical-thinking skills with particular emphasis on consistency, integrity, and fairness
- High attention to detail and comfort reviewing large volumes of similar examples
- Ability to work independently while maintaining consistent evaluation quality
- Collaborative approach to discussing ambiguous cases and refining shared assessment standards
- Excellent written English and professional documentation skills
- Based in the United States, Canada, United Kingdom, Ireland, Australia, or New Zealand
- Authorised to undertake contract work in the relevant country
- No prior formal experience in AI research or model training is required
Benefits:
- Part-time independent contractor engagement
- Fully remote work
- Flexible project scope, workload, timing, and duration depending on project requirements
Similar Jobs
Especialista en Inteligencia Artificial Agente, Banca
Ingeniero de software diseñando soluciones escalables con agentes inteligentes, APIs y automatización. Empresa tecnológica especializada en activos digitales, staffing y fábrica de software.
AI Agent Power User
Part-time AI workflow consultant evaluating ChatGPT and Claude across realistic business processes. Scoring outputs, documenting model behavior, and providing actionable feedback for 24-MAG’s AI projects.
AI Safety Red Teamer
AI Safety Red Teamer stress-testing frontier AI models for Mercor. Designing adversarial prompts, identifying vulnerabilities, and improving model safety with leading AI researchers.
AI Safety Red Teamer
AI Safety Red Teamer adversarially testing frontier AI systems for Mercor. Identifying jailbreaks, hallucinations, and safety failures across high-risk domains.
AI Safety Red Teamer
AI Safety Red Teamer identifying vulnerabilities in frontier AI models for Mercor. Designing adversarial prompts and evaluating high-risk AI behavior.
AI Safety Red Teamer
AI Safety Red Teamer stress-testing frontier AI models for Mercor. Designing adversarial prompts and documenting vulnerabilities across high-risk domains.
Adjunct Instructor, Generative AI
Adjunct Generative AI Instructor teaching adult learners through competency-based online education. Providing AI expertise, personalized support, feedback, and mentorship at Calbright College.
AI Visual Artist
Freelance AI Visual Artist experimenting with generative image, video, music, and text tools. Creating innovative visual concepts for Paintgun’s performance advertising campaigns.
Senior Marketing Strategy, AI Evaluation Specialist
Senior marketing strategist designing evaluation tasks and rubrics for generative AI models. Assessing AI-generated marketing content using brand, growth, and performance expertise.
AI Evaluation Guidelines & Rubric Specialist
AI evaluation specialist developing precise rater guidelines and scoring rubrics for generative AI programmes. Translating complex requirements across finance, legal, retail, insurance, and sports domains.
U.S. Licensed Dermatologist – AI Dermatology Research
U.S.-licensed dermatologist reviewing skin images and AI-generated assessments for 24-MAG LLC’s dermatology research. Establishing clinical guidelines and improving medically grounded AI systems.
AI Safety Red Teamer
AI Safety Red Teamer identifying vulnerabilities in frontier AI systems for Mercor. Designing adversarial prompts and evaluating high-risk model behavior with AI researchers.
AI Safety Red Teamer
AI Safety Red Teamer adversarially testing frontier AI models for Mercor, which partners with AI labs to train and enhance them. Documenting vulnerabilities across high-risk domains.
AI Safety Red Teamer
AI Safety Red Teamer stress-testing frontier AI models for Mercor. Designing adversarial prompts and documenting vulnerabilities to improve AI alignment and safety.
AI Data & Insights Officer
AI Data & Insights Officer turning event and luxury-travel business data into reliable intelligence. Building RAG, dashboards and agent-performance insight for Worldwide Events Group.
AI Data & Insights Officer
AI Data & Insights Officer transforming Worldwide Events Group’s MICE and luxury travel data into reliable intelligence. Building RAG knowledge systems, dashboards and measurable agent performance insights.
Data & AI Analyst
Data & AI Analyst transforming insurance marketing data into actionable insights, dashboards, and AI-powered automation. Building scalable solutions with SQL, BigQuery, Tableau, Python, and emerging AI tools.
AI Safety Red Teamer
AI Safety Red Teamer stress-testing frontier AI models for Mercor. Designing adversarial prompts and documenting vulnerabilities across high-risk domains.
AI Safety Expert, English, Finnish
AI red team expert testing Mercor’s frontier AI models through jailbreaks, prompt injections, and adversarial data generation. Producing reproducible vulnerability reports that improve AI safety.
AI Director
AI Director leading production AI architecture, governance, and client strategy at Futurice, a data-and-AI consultancy. Growing UK capability through consulting, business development, mentoring, and hiring.