AI Benchmark Engineer, Native Language Specialist – Arabic
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Arabic-speaking benchmark engineer designing Terminal-Bench tasks for LILT’s multilingual AI evaluation suite. Building datasets, verifiers, and calibrated tests for coding agents.
Responsibilities:
- Design, build, and validate Terminal-Bench tasks for multilingual software challenges
- Evaluate coding agents
- Build realistic task environments using Arabic datasets and files that remain in the target language
- Find AI failure points in Arabic
- Develop robust reference implementations
- Write reliable, deterministic verifier scripts
- Analyze execution logs and calibrate task difficulty from Easy to Very Hard
- Run standard Terminal-Bench configurations against Haiku, Sonnet, and Opus model tiers
- Participate in a four-layer human quality control process covering creation, human review, calibration review, and audit
- Work alongside automated LLM-based checks to ensure fairness, grammatical accuracy, and benchmark integrity
- Contribute to LILT’s multilingual AI and human-verified services for enterprises, governments, and AI developers
Requirements:
- 5+ years of industry experience in software engineering
- Proven track record at leading technology companies and/or graduation from top-tier engineering universities
- Native or near-native Arabic fluency, with a deep understanding of grammar, register, and phrasing rules
- High English proficiency
- Strong proficiency in Python, standard shell scripting, and data processing
- Extensive experience with Terminal/CLI-based development workflows
- Working familiarity with coding agents
- Deep technical understanding of multilingual text processing pitfalls
- Experience with encoding/decoding robustness and Unicode normalization
- Knowledge of locale-dependent conventions, including collation, casing, and non-Gregorian dates
- Knowledge of text I/O, toolchain interoperability, and safe string operations
- For Arabic, knowledge of bidirectional/RTL handling, font fallbacks, and rendering/typography in UI or artifacts
- Reliable availability and commitment
- Most tasks require a minimum of 2 hours per day or 15–20 hours per week
- As a 1099 contractor, responsibility for own tax obligations
- Candidates must submit an updated CV in English
- Must complete a GenAI assessment
Benefits:
- As an independent contractor, work when you want, as much or as little as you want
- No fixed hours, no check-ins, no micromanaging
- Competitive rates
- Prompt payments
- Access to diverse, innovative projects
- Portfolio and skills development across industries and domains
- Supplemental income opportunity with flexible scheduling
- No benefits such as health insurance, paid time off, or retirement contributions are provided
- Hours are not guaranteed
Similar Jobs
AI Benchmark Engineer, Native Language Specialist – Chinese
AI benchmark engineer building Chinese-language Terminal-Bench tasks for Lilt’s multilingual AI platform. Validating coding agents, reference implementations, and deterministic verifiers.
AI Benchmark Engineer, Native Language Specialist – Chinese
AI benchmark engineer creating Chinese-language Terminal-Bench tasks for Lilt’s multilingual AI platform. Validating coding agents, verifier scripts, and multilingual software robustness.
AI Benchmark Engineer, Native Language Specialist – Arabic
Arabic AI benchmark engineer designing Terminal-Bench tasks for Lilt’s multilingual AI and human-verified language services. Building multilingual assets, verifiers, and calibration workflows for coding agents.
AI Safety Experts – English, Telugu
AI safety red-teamer probing conversational models for Mercor’s human-data AI training projects. Generating vulnerability datasets, attack cases, and reproducible reports to strengthen frontier AI systems.
AI Safety Expert – English, Tamil
AI red-team expert probing conversational models for Mercor, which partners with AI labs to train frontier systems. Generating adversarial data, vulnerability reports, and reproducible attack cases.
AI Safety Expert – English, Kannada
AI safety red teamer probing Mercor’s frontier AI models for jailbreaks, bias, and vulnerabilities. Generating reproducible human-data artifacts for AI labs and enterprise customers.
AI Safety Red Teamer
AI Safety Red Teamer identifying vulnerabilities in frontier AI systems for Mercor. Designing adversarial prompts and evaluating high-risk model behavior.
AI Safety Expert – English, Tamil
AI red team expert probing conversational models for Mercor, which partners with AI labs to train frontier systems. Testing vulnerabilities and producing actionable safety data.
Associate Director – Technology, AI, Digital Solutions – Healthcare
Associate Director leading healthcare technology, AI, and digital transformation engagements for BRG’s healthcare advisory practice. Translating client priorities into technology roadmaps and scalable solutions.
AI Strategist
AI Strategist helping retail and consumer brands identify operational opportunities and deploy applied AI systems. Owning client relationships, business cases, delivery direction, and measurable outcomes across Latin America.
AI Strategist
AI Strategist helping WON redesign retail and consumer business operations with applied AI. Owning client discovery, business cases, delivery direction, and measurable outcomes.
Senior AI Agent Engineer
Senior AI Agent Engineer building Rumor Avenue's new AI product from scratch. Developing Python/FastAPI backend, React interfaces, and production LLM features.
AI Safety Expert, English – Kannada
AI safety red-teaming specialist probing frontier models for Mercor. Generating adversarial test data, vulnerability analyses, and reproducible attack artifacts for AI labs and enterprises.
AI Safety Expert, English, Punjabi
AI safety red teamer probing Mercor-trained AI models through adversarial English and Punjabi testing. Generating vulnerability annotations, attack cases, datasets, and reproducible reports.
AI Safety Expert – English, Odia
AI safety expert red-teaming Mercor’s frontier AI models through adversarial testing and human-data generation. Evaluating vulnerabilities in English and Odia across flexible, fully remote contract projects.
AI Safety Experts – English, Telugu
AI red-teaming contractor probing Mercor’s frontier models with adversarial inputs. Generating human-labeled safety data, vulnerability reports, datasets, and attack cases.
AI Safety Expert – English, Punjabi
AI red-teaming expert probing Mercor customer models with English and Punjabi adversarial inputs. Annotating vulnerabilities and producing reproducible safety datasets and attack reports.
AI Workforce Planning Consultant
Consultoría de planificación laboral con IA para rediseñar organizaciones en Ethikos. Análisis de roles, capacidades y tareas para transformar el trabajo futuro.
External Research Collaborator, Formal Mathematics – AI
External Research Collaborator building AI pipelines that convert mathematics into verified Lean 4 code. Supporting RWS’s mission to unlock global understanding through language and content technology.
AI Enablement Facilitator
AI Enablement Facilitator helping Scale Army, an international staffing agency, sell and deliver AI implementation pods to U.S.-based companies. Facilitating executive workshops, live AI builds, and roadmap closes.