Arabic AI Evaluation Specialist
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Arabic AI Evaluation Specialist role at Welo Data to enhance large language models. Assessing AI performance and ensuring accurate results for Arabic content.
Responsibilities:
- Conduct side-by-side comparisons of AI responses and rate their quality on a 1–5 scale based on established guidelines.
- Design scenario-based and edge-case prompts to evaluate model behavior, including tricky, ambiguous, or incomplete information situations.
- Assess outputs for instruction adherence, factual accuracy, tone, safety, and overall usefulness.
- Develop clear evaluation rubrics and criteria to ensure consistent scoring across tasks.
- Create reliable reference materials (articles, transcripts, reports, etc.) to serve as the source of truth for testing.
- Write well-structured “gold standard” responses that demonstrate the most accurate and helpful answer.
- Identify potential issues such as hallucinations, inconsistencies, or cultural/contextual mismatches.
Requirements:
- Bachelor's degree or equivalent experience in Linguistics, Computational Linguistics, Communications, Technical Writing, or a related analytical field.
- B2 or superior level of English.
- Native fluency in Modern Standard Arabic in Egyptian dialect.
- Strong understanding of the distinction between Fusha and ‘Ammiyya.
- Proven experience in a role involving AI data annotation, content quality review, search quality rating, or prompt engineering.
- Ability to work independently and manage workflows effectively in a remote environment.
- Nice to Have: Multilingual proficiency in one or more Arabic dialects.
- Strong attention to detail and critical thinking to identify hallucinations and bias.
- Familiarity with data annotation platforms and model evaluation tools.
- Experience in prompt engineering, AI evaluation, linguistic QA, or translation is a plus.
- Cultural familiarity with regional norms and high-context communication styles, particularly in the GCC region.
Benefits:
- Limitless Flexibility
- Limitless Growth
- Limitless Support
- Real Impact
Similar Jobs
AI Agent Trainer – Technical Ops Integrator
Software Engineer - AI Trainer developing multi-agent AI systems for businesses while ensuring operational continuity. Involves hands-on technical execution and structured development in an AI-driven environment.
Freelance Energy Sector Expert
Energy Sector Professional at AI startup evaluating energy sector forecasts with strong analytical skills. Work remotely with a YC-backed company focusing on machine learning technologies.
Head of AI Solutioning
Lead AI Solutioning for clients, focusing on business processes and AI architecture development. Engage in technical leadership and team development within a remote work model.
Biology AI Training Expert
Biology expert creating AI training problems reflecting authentic scientific workflows. Designing challenging questions for frontier AI systems in remote work environment.
Biology AI Training Expert
Biology expert designing advanced problems for AI training. Focus on authentic scientific workflows for leading AI models with immediate availability.
Biology AI Training Expert
Biology expert creating challenging, deterministic problems for AI models in a remote role. Focusing on authentic scientific workflows and high standards of rigor and clarity.
Biology AI Training Expert
Biology expert creating challenging problems for frontier AI training data submission and verification. Engaging in computational analysis and experimental reasoning to enhance AI models.
Chemistry AI Training Expert
Chemistry expert creating advanced problems to train AI systems. Role requires strong skills in Python and chemistry knowledge for advanced problem design.
Chemistry AI Training Expert
Experts in Chemistry designing advanced problems for training frontier AI models. Collaborating on high-quality STEM training data with potential project extensions.
Chemistry AI Training Expert
Chemistry AI Training Expert designing advanced deterministic problems for frontier AI models. Focused on high-quality STEM training data for AI labs with rigorous technical explanations.
Chemistry AI Training Expert
Chemistry expert designing advanced deterministic problems for AI model training. Collaborating on high-quality STEM training data for leading AI labs.
Chemistry AI Training Expert
Design difficult chemistry problems for AI training evaluation processes. Collaborate and ensure reproducibility and technical precision in project deliverables.
Chemistry AI Training Expert
Chemistry expert design challenging problems for AI training data. Focusing on real scientific workflows with strong detail and clarity.
Chemistry AI Training Expert
Chemistry expert designing challenging problems for AI training data in STEM. Create original, verifiable tasks with strong computational rigor and precision.
Biology AI Training Expert
Biology expert creating advanced problems for AI systems. Involves computational analysis, Python usage, and ensuring technical clarity for quality scientific data.
Biology AI Training Expert
Biology expert creating deterministic problems for AI models, improving scientific reasoning in biology. Seeking strong Python and biological analysis skills, fully remote role.
Computer Science AI Training Expert
Computer Science expert needed to design challenging, verifiable problems for AI training. Work involves creating advanced problems in algorithms, machine learning, and computational methods.
Computer Science AI Training Expert
Computer Science expert designing challenging, original computational problems for AI training. Creating tasks that enhance model reasoning with verifiable solutions in a remote capacity.
Computer Science AI Training Expert
Design and create challenging computational problems for AI training in Computer Science. Collaborate to improve technical reasoning in leading AI models.
Computer Science AI Training Expert
Design advanced computational problems and tasks for AI systems. Collaborate to enhance AI model performance through challenging problem sets and verified solutions.