Software, AI, IT, Data Evaluator
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
AI Evaluator reviewing documents, spreadsheets, and slide decks for accuracy and domain quality. Helping Mercor’s AI lab partners train and enhance frontier models.
Responsibilities:
- Review and assess AI-generated work products, including documents, spreadsheets, and slide decks
- Apply deep subject-matter expertise to grade AI-generated outputs
- Evaluate AI-generated artifacts against domain-specific quality rubrics
- Identify factual, aesthetic, and presentation errors
- Provide clear, structured written feedback
- Work on projects focused on training and enhancing AI systems
- Collaborate with leading researchers
Requirements:
- 5+ years of relevant professional experience in Software / AI / IT / data
- Native or professional fluency in English
- Highly proficient in Microsoft Office and Google Workspace
- Especially proficient in Google Slides and PowerPoint
- Advanced degree (Master's or higher) from a reputable institution (preferred, nice to have)
- Must be unable to require H1-B or STEM OPT support
Benefits:
- Fully remote role
- Flexible schedule / work on your own schedule
- Weekly payments via Stripe or Wise
- Competitive pay
- Referral bonus of up to $480 per successful referral
- Reasonable accommodations upon request
- Potentially extendable project engagement
Similar Jobs
Director, Information Technology – Enterprise AI
Director leading IT, cybersecurity, and enterprise AI strategy for Vehlo, an auto repair technology and financial solutions provider. Governing security, AI platforms, infrastructure, and technology operations.
Senior Director Analyst – Integration and APIs, AI
Senior Gartner analyst shaping AI-era integration and API research. Advising enterprise leaders on MCP, AI gateways, and integration strategy.
Senior Director, Analyst – Tech CEO Business and Strategy Advisor on AI Orchestration
Senior Gartner analyst advising Texas-based technology CEOs on AI Orchestration growth, market dynamics, and innovation. Creating research, frameworks, forecasts, and presentations for Gartner clients and events.
Senior Director, Analyst – Healthcare Providers, RCM & Healthcare AI Technology
Senior Director shaping Gartner insights on healthcare providers' U.S. revenue cycle management, coding, and AI technologies. Advising executives and clients through research, presentations, and strategic guidance.
AI Safety Red Teamer
AI Safety Red Teamer stress-testing frontier AI models for Mercor, which partners with AI labs to train and enhance AI systems. Identifying vulnerabilities and improving model safety.
Bilingual Dutch STEM Expert – AI Safety
PhD STEM expert evaluating Dutch-language scientific AI responses for Mercor. Writing specialist prompts and annotating model safety, accuracy, and helpfulness.
Bilingual Portuguese Generalist Expert – AI Safety
Bilingual Portuguese Generalist evaluating sensitive-topic AI model safety for Mercor. Writing prompts, classifying conversations, and flagging adversarial patterns.
Bilingual Danish STEM Expert – AI Safety
Bilingual Danish PhD STEM expert writing prompts and evaluating scientific responses. Improving AI safety for Mercor’s frontier-model training projects.
Bilingual German Generalist Expert – AI Safety
Bilingual German Generalist evaluating sensitive-topic AI behavior for Mercor’s frontier-model training projects. Writing German prompts, classifying conversations, and flagging adversarial patterns.
Bilingual Korean Generalist Expert – AI Safety
Bilingual Korean Generalist Expert evaluating sensitive-topic handling in Korean for Mercor’s frontier AI training projects. Writing prompts, classifying conversations, and flagging adversarial patterns.
Bilingual Danish Generalist Expert – AI Safety
Bilingual Danish Generalist Expert evaluating sensitive AI-model behavior for Mercor. Writing prompts, classifying conversations, and flagging adversarial patterns in frontier-model training projects.
Bilingual Russian Generalist Expert – AI Safety
Bilingual Russian Generalist evaluating sensitive-topic prompts and conversations for Mercor. Applying language expertise to train safer frontier AI models.
Bilingual Croatian STEM Expert – AI Safety
Bilingual Croatian PhD scientist evaluating scientific AI responses and writing safety prompts. Supporting Mercor’s frontier-model training through expert technical annotation.
AI Safety Red Teamer
AI Safety Red Teamer identifying vulnerabilities in frontier AI models for Mercor’s AI lab partnerships. Designing adversarial prompts and evaluating high-risk model behavior.
AI Safety Expert – English, Portuguese
AI safety red teamer probing conversational models for jailbreaks, prompt injections, and systemic risks. Producing human-data artifacts that help Mercor customers build safer frontier AI.
AI Safety Experts – English, Bengali
AI safety red team expert probing Mercor’s partner AI models for vulnerabilities. Generating reproducible adversarial data, reports, and attack cases to improve model safety.
AI Safety Red Teamer
AI Safety Red Teamer stress-testing frontier AI models for Mercor’s AI training partnerships. Identifying vulnerabilities and improving model alignment, robustness, and safety.
Principal AI Search – Conversation Architect
Principal AI architect building conversational search and commerce orchestration for OleOle’s global football super app. Designing LLM, agent, retrieval, and recommendation systems across real-time sports services.
AI Workflow Engineer
AI workflow engineer building Python and GenAI automation for BlueCross BlueShield of Tennessee. Supporting Treasury and Internal Audit workflow modernization, governance, training, and adoption.
AI Prompt Engineer
Prompt Systems Engineer designing and evaluating AI agents for Noodle, a higher-education strategy, services, and technology partner. Improving enrollment, learning, and student-support experiences.