Machine Learning Engineering Manager – LLM Serving, Infrastructure
Employment Information
Job Description
Machine Learning Manager developing innovative LLM serving infrastructure for Spotify's Personalization team. Leading a high-performance engineering team to enhance user experience in music and podcasts.
Responsibilities:
- Lead a high-performing engineering team to develop, build, and deploy a high-scale, low-latency LLM Serving Infrastructure.
- Drive the implementation of a unified serving layer to support multiple LLM models and inference types (batch, offline eval flows and real-time/streaming).
- Lead all aspects of the development of the Model Registry for deploying, versioning, and running LLMs across production environments.
- Ensure successful integration with the core Personalization and Recommendation systems to deliver LLM-powered features.
- Define and champion standardized technical interfaces and protocols for efficient model deployment and scaling.
- Establish and monitor the serving infrastructure's performance, cost, and reliability, including load balancing, autoscaling, and failure recovery.
- Collaborate closely with data science, machine learning research, and feature teams (Autoplay, Home, Search, etc.) to drive the active adoption of the serving infrastructure.
- Scale up the serving architecture to handle hundreds of millions of users and high-volume inference requests for internal domain-specific LLMs.
- Drive Latency and Cost Optimization: partner with SRE and ML teams to implement techniques like quantization, pruning, and efficient batching to minimize serving latency and cloud compute costs.
- Develop Observability and Monitoring: build dashboards and alerting for service health, tracing, A/B test traffic, and latency trends to ensure consistency to defined SLAs.
- Contribute to Core LPM Serving: focus on the technical strategy for deploying and maintaining the core Large Personalization Model (LPM).
Requirements:
- 5+ years of experience in software or machine learning engineering, with 2+ years of experience managing an engineering team.
- Hands-on with ML Engineering: you have deep expertise in building, scaling, and governing high-quality ML systems and datasets, including defining data schemas, handling data lineage, and implementing data validation pipelines (e.g., HuggingFace datasets library or similar internal systems).
- Deep technical background in building and operating large-scale, high-velocity Machine Learning/MLOps infrastructure, ideally for personalization, recommendation, or Large Language Models (LLMs).
- Proven track record to drive complex projects involving multiple partners and federated contribution models ("one source of truth, many contributors").
- Expertise in designing robust, loosely coupled systems with clean APIs and clear separation of concerns (e.g., distinguishing between fast dev-time tools and rigorous production-like systems).
- Experience integrating evaluation and testing into continuous integration/continuous deployment (CI/CD) pipelines to enable rapid 'fork-evaluate-merge' developer workflows.
- Solid understanding of experiment tracking and results visualization platforms (e.g., MLFlow, custom UIs).
- A pragmatic leader who can balance the need for speed with progressive rigor and production fidelity.
Benefits:
- health insurance
- six month paid parental leave
- 401(k) retirement plan
- monthly meal allowance
- 23 paid days off
- 13 paid flexible holidays
- paid sick leave
Related Jobs
Engineering Manager
DocPlanner
Engineering Manager leading a team of ~10 at Feegow, a health-tech company. Responsible for managing teams, ensuring high-quality delivery and strategic direction.
Engineering Manager
Gravie
Engineering Manager leading high-performing engineering teams at Gravie to enhance consumer-centric healthcare solutions. Fostering collaboration and guiding technical direction for software delivery.
Engineering Manager – Cross Retailer Search
Instacart
Engineering Manager overseeing Cross Retailer Search team at Instacart. Leading backend systems and ML-driven projects for enhanced customer engagement and retention.
VP of Engineering
Taekus
VP of Engineering leading engineering efforts in a high-growth fintech company. Collaborate closely with CEO and executive team on technical vision and delivery excellence.
Manager, Software Engineering – Node
DriveTime
Manager, Software Engineering leading a team of software engineers at GoFi, an AI-centric auto finance company. Responsible for building applications and mentoring engineers in a collaborative environment.
Machine Learning Practice Leader – Engineering Manager
Provectus
Senior technical leader managing and growing ML engineering teams and delivering AI-driven projects. Role combines machine learning expertise with strategic leadership in fully remote setup.
Engineering Manager
Celigo
Engineering Manager at Celigo leading platform teams to deliver scalable application integrations. Focused on team management and collaboration with cross-functional stakeholders.
Rail Engineering Director
AECOM
Rail Engineering Director overseeing planning, sustainability, and engineering from conception through construction. Collaborating with clients and stakeholders on large-scale rail projects based in the United States.
Engineering Manager
ActBlue
Engineering Manager at ActBlue leading a team through the evolution of a new strategic platform. Focused on fostering user trust in a regulated environment while building collaborative relationships.
Manager, Software Engineering
Natera
Engineering Manager leading a skilled team in software development for Natera, a biotech company specializing in oncology. Ensuring project delivery and technical excellence for critical applications.
Data Engineering Manager, Integrations
SmarterDx
Data Engineering Manager leading a data engineering team at SmarterDx for healthcare data integrations and improving operational efficiency.
Senior Engineering Manager – Risk
Mercury
Senior Engineering Manager leading engineering teams for risk and compliance systems at fintech company Mercury. Driving system design and operational excellence for business banking applications.
Engineering Manager, Invoicing Platform
Instacart
Engineering Manager leading a team to build the invoicing platform at Instacart. Ensuring accurate invoicing data for billions in transactions while collaborating with cross-functional teams.
Senior Manager, Software Engineering
Addium
Senior Manager leading software engineering team at Addium, committed to sustainable water use. Overseeing QA, resources, and offshore collaboration with a strategic approach.
Head of Engineering – Payments
DEUNA
As Head of Engineering, lead the technical vision for payments platform at DEUNA. Oversee engineers and technical leads in building scalable payment systems.
Senior Engineering Manager
LearnLux Financial Wellbeing
Leading engineering initiatives at LearnLux, a platform for financial wellbeing. Overseeing DevOps and AI Platform teams to ensure reliable systems and drive innovation.
Engineering Manager – Tech Lead
redbee
Engineering Manager at redbee building high-quality technology products with a focus on innovation and teamwork. Leading multidisciplinary teams and driving project execution in digital product development.
Engineering Manager
NBCUniversal
Engineering Manager overseeing implementation and optimization of SAP BTP solutions for Comcast. Leading engineering teams to design and deploy cloud-based applications in compliance with business objectives.
Senior Manager, Software Engineering
NBCUniversal
Senior Manager, Backend Engineering leading re-platforming of backend services for VERSANT Media. Directing modernization of APIs and microservices while mentoring a team of engineers.
Engineering Manager – Internationalisation
Canva
Engineering Manager leading cross-functional backend and infrastructure team at Canva. Focusing on localised features, team coaching, and technical roadmap development.