AI Voice Evaluation Specialist
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
AI voice evaluator comparing two conversational AI models for Innodata, a global data engineering company. Recording interactions, assessing quality, categorizing errors, and documenting model preferences.
Responsibilities:
- Review an assigned scenario and roleplay a natural conversation with two different AI models.
- Conduct comparable conversations with Model A and Model B, keeping the scenario, conversational approach, and overall interaction as consistent as possible.
- Maintain approximately the same number of conversational turns with each model to support a fair comparison.
- Record your voice during each interaction and pay close attention to the quality and behavior of the model's responses.
- Evaluate each model across five defined evaluation dimensions.
- Identify and categorize relevant error clusters that may occur in the audio or conversation.
- Compare the performance of Model A and Model B based on the evaluation criteria and your observations.
- Select the model that demonstrated the stronger overall performance according to the defined evaluation framework.
- Write a detailed rationale explaining your final preference, referencing specific examples and observations from both conversations.
- Apply evaluation guidelines consistently across different scenarios and model interactions
Requirements:
- Bachelors Degree
- Strong attention to detail and ability to notice subtle differences in conversational quality.
- Excellent listening and comprehension skills.
- Strong written communication skills, particularly the ability to explain observations clearly and objectively.
- Ability to follow detailed evaluation guidelines consistently.
- Comfort speaking naturally and roleplaying different conversational scenarios.
- Ability to compare two interactions fairly without allowing personal preferences to influence the evaluation.
- Strong critical-thinking and analytical skills.
- Reliability and consistency when completing structured evaluation tasks.
- Familiarity with AI assistants, voice-based AI, or conversational systems
Similar Jobs
AI Safety Expert – English, Kannada
AI safety red-team expert probing frontier AI models for Mercor. Testing vulnerabilities, generating human data, and producing reproducible safety artifacts for AI labs and enterprises.
AI Safety Expert, English – Danish
AI red-teaming expert probing Mercor clients’ frontier AI models for jailbreaks, bias, and misuse. Generating reproducible safety data, attack cases, and vulnerability reports.
AI Safety Expert – English, Punjabi
AI safety red teamer probing conversational AI models for Mercor, which partners with AI labs to train frontier systems. Generating adversarial datasets, vulnerability reports, and attack cases.
AI Safety Expert – English, Punjabi
AI red-teaming contractor probing conversational models for Mercor, which partners with AI labs to train frontier systems. Generating vulnerability data, attack cases, and safety insights.
Workflow & AI Agent Engineer
Workflow and AI Agent Engineer supporting clients of Chatarmin’s B2B WhatsApp marketing SaaS. Owning onboarding, support, API-driven solutions, and product feedback loops.
AI Safety Expert – English, Telugu
AI safety red-teamer probing conversational models for Mercor. Generating adversarial datasets and reproducible vulnerability reports to help AI labs and enterprises build safer systems.
AI Systems Innovation Consultant
AI Systems Innovation Consultant für Aivisorys KI-Beratung. Entwicklung und Umsetzung von KI-Strategien, Workshops und Lösungen für Kunden.
Co-Founder, Tech Partner – AI, Open Source
Technical co-founder designing open-source AI and data infrastructure for ESG, governance, and process intelligence. Building architecture, integrations, and LLM-powered applications within Estonia’s startup ecosystem.
Working Student – AI Sales
Werkstudent im B2B-Vertrieb für eine Berliner KI-Agenten- und Sprachautomatisierungsplattform. Generierung und Qualifizierung von Leads durch Cold Calling, E-Mail-Outreach und Terminvereinbarung.
AI Safety Expert – English, Gujarati
AI red-teaming contractor probing conversational models for Mercor, which partners with leading AI labs to train frontier systems. Generating vulnerability data, attack cases, and reproducible safety reports.
AI Safety Expert – English, Telugu
AI red-teaming experts probing Mercor’s frontier AI systems in English and Telugu. Testing conversational models for vulnerabilities and producing actionable safety data.
AI Safety Expert – English, Tamil
AI red-teaming contractor probing Mercor customer models for jailbreaks, bias, and misuse. Generating reproducible safety data for frontier AI labs and enterprises.
AI Safety Expert – English, Assamese
AI safety experts red-teaming conversational AI models for Mercor, which partners with leading AI labs to train frontier systems. Generating vulnerability data, attack cases, and reproducible safety reports.
AI Safety Expert – English, Gujarati
AI red-teaming expert probing conversational models for Mercor, which partners with AI labs to train frontier systems. Generating adversarial data, vulnerability reports, and reproducible attack cases.
AI Safety Expert, English, Gujarati
AI red-teaming expert probing Mercor customer models for jailbreaks, bias, and misuse. Generating reproducible human-data artifacts that improve frontier AI safety.
AI Safety Expert – English, Tamil
AI safety red teamer probing conversational AI models for Mercor, which partners with leading AI labs to train frontier systems. Generating vulnerability data, attack cases, and reproducible reports.
AI Safety Experts, English – Marathi
AI red-teaming experts testing Mercor’s frontier models through adversarial English and Marathi analysis. Probing vulnerabilities and producing datasets that make customer AI systems safer.
AI Safety Expert – English, Bengali
AI safety red teamer probing conversational AI models for Mercor, which partners with AI labs to train frontier systems. Generating vulnerability data and reproducible attack cases.
AI Safety Expert – English, Urdu
AI safety red teamer probing Mercor-partnered frontier AI models for vulnerabilities. Testing conversational systems and generating actionable human data in English and Urdu.
AI Safety Expert – English, Bengali
AI safety red teamer probing frontier AI models for Mercor, which partners with leading labs and enterprises to train models. Generating vulnerability data and reproducible attack artifacts.