Senior Machine Learning Engineer, LLM Inference Optimization
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior ML Engineer optimizing LLM inference services for Nebius’s AI cloud platform. Improving latency, throughput, GPU utilization, memory efficiency, and cost per token.
Responsibilities:
- Own optimization work for specific model families, customer endpoints, or serving backends
- Run engine comparisons and recommend practical serving configurations for specific workloads
- Debug model quality or performance regressions during production rollouts
- Optimize LLM and VLM endpoints for latency, throughput, memory efficiency, GPU utilization, quality, and cost per token
- Deploy, configure, benchmark, and extend inference engines such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, or similar systems
- Build and productionize model-compression workflows, including quantization, quantization-aware training, distillation, low-bit serving, and accuracy recovery
- Implement or integrate speculative decoding, draft-model approaches, KV-cache optimization, prefix caching, chunked prefill, continuous batching, and disaggregated prefill/decode serving
- Build reproducible benchmark harnesses for TTFT, TPOT, tokens per second per GPU, p95/p99 latency, GPU memory, reliability, and cost per token
- Partner with GPU kernel engineers and platform engineers to diagnose bottlenecks across model code, kernels, runtime, scheduler, gateway, and cluster layers
- Write clear design docs, performance reports, rollout plans, and customer-facing technical explanations
Requirements:
- Strong Python and PyTorch engineering skills
- Hands-on experience deploying or optimizing LLM, VLM, or high-throughput transformer inference systems
- Practical knowledge of at least one modern inference stack such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, Ray Serve, KServe, or equivalent internal systems
- Strong understanding of transformer inference bottlenecks, including KV cache, attention, memory bandwidth, batching, parallelism, and long-context serving
- Ability to reason quantitatively about latency, throughput, quality, utilization, and cost tradeoffs
- Strong communication skills and ability to collaborate with research, kernel, infrastructure, product, and customer teams
- Experience with quantization-aware training, post-training quantization, FP8, INT8, INT4, NVFP4, MXFP4, AWQ, GPTQ, SmoothQuant, or related techniques
- Experience with distillation, speculative decoding, EAGLE, Medusa, multi-token prediction, or other inference acceleration methods
- Experience with agentic workloads, including tool calling, structured outputs, streaming APIs, high concurrency, and multi-step orchestration
- CUDA or Triton familiarity
- Open-source contributions to vLLM, SGLang, TensorRT-LLM, FlashInfer, LMCache, PyTorch, Triton, Ray, KServe, or related projects
Benefits:
- Competitive compensation
- Career growth and learning opportunities
- Flexibility and ownership
- Collaborative and innovative culture
- Opportunity to work on impactful AI projects
- International environment and talented teams
Similar Jobs
Associate Staff Engineer – Conversational AI Developer
Kore.ai conversational AI engineer building chatbots, AI agents, and Agentic AI solutions for Nagarro. Integrating LLMs, enterprise applications, and Vector Databases across South Asia.
Senior Staff Engineer, Generative AI
Senior Staff Engineer architecting and productionizing scalable GenAI and Agentic AI solutions for Nagarro’s digital product engineering services. Building Python, FastAPI, RAG, and cloud-based AI systems.
Staff Engineer, Generative AI
Staff Engineer architecting and productionizing scalable GenAI and Agentic AI solutions for Nagarro’s digital product engineering services. Building Python/FastAPI systems across Azure and AWS.
Staff Engineer, Generative AI
Staff Engineer architecting and coding scalable Generative AI and Agentic AI solutions at Nagarro, a digital product engineering company. Deploying enterprise AI systems on Azure or AWS.
AI Engineer – NLP, Large Language Models Specialist
AI Engineer developing NLP and LLM applications for DOTSOFT, a Greek ICT innovator. Building assistants, chatbots, semantic search, and multilingual AI solutions.
Senior Machine Learning Engineer – Generative AI
Senior Machine Learning Engineer building generative AI applications for The Home Depot. Developing LLM, RAG, and production AI solutions.
AI/LLM Engineer, PL
Engenheiro de AI/LLM liderando aplicações inteligentes com OpenAI, Anthropic e pipelines RAG. Construção de chatbots, copilots e agentes autônomos escaláveis e confiáveis.
Senior AI/LLM Engineer
Engenheiro(a) sênior de IA/LLM desenvolvendo chatbots, copilots e agentes autônomos. Integrando modelos, pipelines RAG e observabilidade para produção confiável e escalável.
Senior LLM Engineer
Senior LLM Engineer fine-tuning and deploying Azure LLMs for Molex engineering design platforms. Building RAG, agentic workflows, evaluation harnesses, and optimized inference.
Senior AI Engineer – Generative AI, Pharmacovigilance
AI Engineer building Generative AI, RAG, and Agentic AI solutions for EVERSANA’s pharmacovigilance technology team. Developing compliant drug-safety monitoring and case-processing applications.
AI/ML Engineer – LLM & AI Harness Engineering
AI/ML Engineer building Python-based LLM, RAG, and agent harnesses for engineering data and workflows at a global technology company. Supporting automation, analysis, and product-development decisions.
Senior AI Engineer, LLM, Agents
Fictional MoralesHR Senior AI Engineer building an unspecified LLM agent platform. Designing RAG, model fine-tuning, latency, and evaluation systems that do not exist.
Senior AI Solutions Engineer – Generative AI, Agentic AI
AI Solutions Engineer building production agents and workflows for EVERSANA’s healthcare commercialization services. Integrating Vertex AI, Claude, Gemini, Salesforce, MuleSoft, and Java systems for Patient Services.
Senior Applied AI Engineer, GenAI, LLM Evaluation
Senior AI & Data Engineer developing AI-powered data products for Acuity Analytics. Building scalable data pipelines and evaluating LLMs while collaborating with Data Scientists and Engineers.
AI Infrastructure Co-Founder, CSO
Co-Founder / CSO role at EWOR GmbH for ambitious entrepreneurs in AI Infrastructure startups. Involves personal development and direct mentoring for building high-revenue startups.
AI Engineer, Generative AI
AI Engineer developing and deploying generative AI applications with Amazon Bedrock. Focused on creating production-ready AI platforms for non-technical users using natural language.
Generative AI Engineer – LATAM Candidates Only
Generative AI Engineer developing advanced AI solutions leveraging LLMs at Talentus Global. Collaborating with various teams to integrate and improve AI applications in cloud environments.
Especialista I – Engenheiro de IA Generativa, Agentes
Implement independent reasoning flows and integrate via MCP for Grupo Boticário. Focus on developing tools and ensuring performance in AI operations.
Senior NLP/LLM Engineer
Senior NLP/LLM Engineer exploring and analyzing LLM capabilities while collaborating with cross-functional teams at Social Discovery Group. Enhancing AI models' effectiveness and optimizing their performance.
Senior Director, AI Infrastructure
Sr. Director of AI Infrastructure at ARMRA connecting business systems into a unified intelligent automation layer. Owning scalable architecture and AI playbook development across all functions.