AI Benchmark Engineer, Native Language Specialist – Arabic
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Arabic AI benchmark engineer designing Terminal-Bench tasks for Lilt’s multilingual AI and human-verified language services. Building multilingual assets, verifiers, and calibration workflows for coding agents.
Responsibilities:
- Design, build, and validate Terminal-Bench tasks for multilingual software challenges
- Evaluate coding agents
- Build realistic task environments using datasets and files in Arabic, keeping assets in the target language
- Find AI failure points through prompting and translation work in Arabic
- Develop robust reference implementations
- Write reliable, deterministic verifier scripts
- Analyze execution logs and calibrate task difficulty from Easy to Very Hard
- Run standard Terminal-Bench configurations against Haiku, Sonnet, and Opus model tiers
- Participate in four-layer human quality control: creation, human review, calibration review, and audit
- Work alongside automated LLM-based checks to ensure fairness, grammatical accuracy, and benchmark integrity
- Support multilingual AI and human-verified language services for enterprises, governments, and AI developers
Requirements:
- 5+ years of industry experience in software engineering
- Proven track record at leading technology companies and/or graduation from top-tier engineering universities
- Native or near-native fluency in Arabic with deep understanding of grammar, register, and phrasing rules
- High English proficiency
- Strong proficiency in Python, standard shell scripting, and data processing
- Extensive experience with Terminal/CLI-based development workflows
- Working familiarity with coding agents
- Deep technical understanding of multilingual text-processing pitfalls
- Experience with encoding/decoding robustness and Unicode normalization
- Knowledge of locale-dependent conventions, including collation, casing, and non-Gregorian dates
- Experience with text I/O, toolchain interoperability, and safe string operations
- For Arabic: knowledge of bidirectional/RTL handling, font fallbacks, and rendering/typography in UI or artifacts
- Reliable availability and commitment to on-time delivery
- Most tasks require availability of at least 2 hours per day or 15–20 hours per week
- Ability to submit an updated CV in English
- Must complete a GenAI assessment
- Contractors cannot be engaged in regions subject to international embargoes or sanctions
- Responsible for own tax obligations
Benefits:
- Flexible schedule; work when you want, as much or as little as you want
- No fixed hours, check-ins, or micromanaging
- Competitive rates
- Prompt payments
- Access to diverse, innovative projects
- Portfolio-building and professional skill development across industries and domains
- Global community of linguists, subject matter experts, and language professionals
- Remote work from anywhere, any time you want
- No health insurance, paid time off, or retirement contributions are provided
- Hours are not guaranteed
Similar Jobs
AI Benchmark Engineer, Native Language Specialist – Chinese
AI benchmark engineer building Chinese-language Terminal-Bench tasks for Lilt’s multilingual AI platform. Validating coding agents, reference implementations, and deterministic verifiers.
AI Benchmark Engineer, Native Language Specialist – Chinese
AI benchmark engineer creating Chinese-language Terminal-Bench tasks for Lilt’s multilingual AI platform. Validating coding agents, verifier scripts, and multilingual software robustness.
AI Benchmark Engineer, Native Language Specialist – Arabic
Arabic-speaking benchmark engineer designing Terminal-Bench tasks for LILT’s multilingual AI evaluation suite. Building datasets, verifiers, and calibrated tests for coding agents.
AI Safety Experts – English, Telugu
AI safety red-teamer probing conversational models for Mercor’s human-data AI training projects. Generating vulnerability datasets, attack cases, and reproducible reports to strengthen frontier AI systems.
AI Safety Expert – English, Tamil
AI red-team expert probing conversational models for Mercor, which partners with AI labs to train frontier systems. Generating adversarial data, vulnerability reports, and reproducible attack cases.
AI Safety Expert – English, Kannada
AI safety red teamer probing Mercor’s frontier AI models for jailbreaks, bias, and vulnerabilities. Generating reproducible human-data artifacts for AI labs and enterprise customers.
AI Safety Red Teamer
AI Safety Red Teamer identifying vulnerabilities in frontier AI systems for Mercor. Designing adversarial prompts and evaluating high-risk model behavior.
AI Safety Expert – English, Tamil
AI red team expert probing conversational models for Mercor, which partners with AI labs to train frontier systems. Testing vulnerabilities and producing actionable safety data.
Associate Director – Technology, AI, Digital Solutions – Healthcare
Associate Director leading healthcare technology, AI, and digital transformation engagements for BRG’s healthcare advisory practice. Translating client priorities into technology roadmaps and scalable solutions.
AI Strategist
AI Strategist helping retail and consumer brands identify operational opportunities and deploy applied AI systems. Owning client relationships, business cases, delivery direction, and measurable outcomes across Latin America.
AI Strategist
AI Strategist helping WON redesign retail and consumer business operations with applied AI. Owning client discovery, business cases, delivery direction, and measurable outcomes.
Senior AI Agent Engineer
Senior AI Agent Engineer building Rumor Avenue's new AI product from scratch. Developing Python/FastAPI backend, React interfaces, and production LLM features.
AI Safety Expert, English – Kannada
AI safety red-teaming specialist probing frontier models for Mercor. Generating adversarial test data, vulnerability analyses, and reproducible attack artifacts for AI labs and enterprises.
AI Safety Expert, English, Punjabi
AI safety red teamer probing Mercor-trained AI models through adversarial English and Punjabi testing. Generating vulnerability annotations, attack cases, datasets, and reproducible reports.
AI Safety Expert – English, Odia
AI safety expert red-teaming Mercor’s frontier AI models through adversarial testing and human-data generation. Evaluating vulnerabilities in English and Odia across flexible, fully remote contract projects.
AI Safety Experts – English, Telugu
AI red-teaming contractor probing Mercor’s frontier models with adversarial inputs. Generating human-labeled safety data, vulnerability reports, datasets, and attack cases.
AI Safety Expert – English, Punjabi
AI red-teaming expert probing Mercor customer models with English and Punjabi adversarial inputs. Annotating vulnerabilities and producing reproducible safety datasets and attack reports.
AI Workforce Planning Consultant
Consultoría de planificación laboral con IA para rediseñar organizaciones en Ethikos. Análisis de roles, capacidades y tareas para transformar el trabajo futuro.
External Research Collaborator, Formal Mathematics – AI
External Research Collaborator building AI pipelines that convert mathematics into verified Lean 4 code. Supporting RWS’s mission to unlock global understanding through language and content technology.
AI Enablement Facilitator
AI Enablement Facilitator helping Scale Army, an international staffing agency, sell and deliver AI implementation pods to U.S.-based companies. Facilitating executive workshops, live AI builds, and roadmap closes.