Senior AI Development Lead

Posted 6hrs ago

Employment Information

Education
Salary
Experience
Job Type

Report this job

Job expired or something wrong with this job?

Job Description

Senior AI Development Lead building and evaluating AI workflows for Airbnb’s global marketplace. Shipping measurable automations and features with product, engineering, business, Legal and Policy partners.

Responsibilities:

  • Drive AI workflows and features from design to launch, defining quality standards and measurement plans with engineering, product and business partners
  • Prioritize projects by business impact and make trade-offs with the manager and business and product leads
  • Map existing processes and determine how AI can perform them and where human judgment is needed
  • Design and build AI workflows, including prompts, pipelines and tooling
  • Monitor quality and drift after launch
  • Design evaluation frameworks, build test sets, define accuracy and agreement metrics, and calibrate LLM judges against human reviewers
  • Partner with engineers on architecture and tooling decisions and pressure-test estimates and scope
  • Convert unclear guidelines into consistent reviewer criteria with product, Legal and Policy
  • Create labeling guidelines and human-verified labels with subject matter experts
  • Run labeling at scale with vendor labeling teams and audit results using agreement metrics and spot checks
  • Define launch quality standards for AI features and provide evidence and go/no-go recommendations
  • Build reusable evaluation and workflow components, document learnings and drive adoption across teams
  • Keep methods current as models, tools and policies change and recommend when to retire outdated approaches

Requirements:

  • 8+ years in product or program management, analytics, consulting or applied AI
  • Proven record of taking technical programs from idea to production and owning results such as backlog, cycle time or launch speed
  • Strong SQL skills
  • Strong knowledge of accuracy, precision and recall, sampling and reviewer agreement
  • Experience shipping an AI workflow, agent or automation relied upon by others, with demonstrable before-and-after metrics
  • Technical fluency sufficient to pressure-test engineering estimates, understand prompts, pipelines and evaluations, and assess workarounds
  • Experience making messy processes measurable and evaluating business outcomes
  • Ability to assess qualitative feedback and reviewer disagreements
  • Clear writing and ability to work with sales, operations, PMs and engineers
  • Willingness to be the first person on a problem and try things directly
  • Must live in a state where Airbnb, Inc. has a registered entity
  • Nice to have: hands-on Python and API experience
  • Nice to have: LLM evaluation experience
  • Nice to have: experimentation and measurement experience
  • Nice to have: trust and safety, content moderation, customer support automation, annotation operations or other human-in-the-loop work at scale
  • Nice to have: background in a two-sided marketplace, travel or another operations-heavy business

Benefits:

  • Bonus eligibility
  • Equity eligibility
  • Benefits eligibility
  • Employee Travel Credits
  • Occasional work at an Airbnb office or attendance at offsites, as agreed with the manager
  • Reasonable accommodations throughout the recruitment process