Principal Software Engineer – E2E Performance, Goodput
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Principal engineer optimizing NVIDIA GPU performance for hyperscale cloud customers. Driving profiling, benchmarking, and systemic improvements across CSP deployments.
Responsibilities:
- Drive performance characterization work streams with engineering teams of key CSP/hyperscale customers, ensuring understanding of platform performance expectations, profiling methodology, and tuning options for specific workloads
- Gather and synthesize CSP performance feedback, identify gaps between expected and actual throughput, and champion optimization priorities to NVIDIA's CUDA, NCCL, driver, and firmware teams
- Ensure open-source performance and stress tools such as STREAM, GPU Burn, and GPU BLAST are updated and validated for the latest NVIDIA rack-scale systems, GPU architectures, and CPU platforms
- Work with CSPs to ensure their performance and validation tooling reflects the latest GPU capabilities, memory hierarchy changes, and platform-specific tuning parameters
- Conduct cross-CSP performance comparison and pattern analysis to identify configuration, software, or workload differences explaining deployment performance gaps
- Collaborate with CSPs to prepare profiling infrastructure, benchmark harnesses, and configuration validation for deployment milestones
- Define test strategies and tooling requirements for performance validation for NVIDIA internal certification and customer acceptance
Requirements:
- 15+ years of experience in systems performance engineering, ideally in GPU/HPC/ML infrastructure
- BS or MS in Computer Science, Computer Engineering, or related field (or equivalent experience)
- Proficiency in GPU workload profiling: nsight systems, nsight compute, DCGM metrics, or equivalent instrumentation
- Understanding of distributed training performance dynamics: computation/communication overlap, pipeline bubbles, memory bandwidth utilization, collective efficiency
- Statistical methods for performance analysis: regression detection, confidence intervals, A/B comparison at scale
- Understanding of how the full software stack impacts performance: driver overhead, collective algorithm selection, memory allocation, scheduling, firmware power management
- Strong data analysis and visualization skills (Python, pandas, dashboards)
- Ability to communicate performance findings to both deep technical audiences and executive leadership
- Demonstrated success influencing multiple engineering teams to prioritize performance improvements
- Experience profiling and optimizing distributed training at 1000+ GPU scale (Megatron-LM, DeepSpeed, FSDP)
- Background in ML infrastructure performance at a CSP/hyperscaler
- Familiarity with NVIDIA platforms (DGX, HGX, NVLink topology) and profiling tools
- Experience building automated performance regression detection systems for production environments
- Understanding of inference workload performance dynamics (vLLM, TensorRT-LLM, SGLang, continuous batching)
Benefits:
- Equity
- Benefits
Similar Jobs
Software Developer
Software Developer building Salesforce applications and integrations for Tripoint Solutions’ federal higher-education platform. Supporting Education Cloud migration, Agentforce, security, and production operations.
Lead Software Engineer
Team Lead guiding developers building GRC web applications for SBS CyberSecurity, a cybersecurity firm. Leading delivery, code quality, hiring, mentoring, and AI-assisted development.
Senior Full Stack Developer – Frontend Oriented
Principal Engineer directing architecture for Terrific Live, a social commerce platform for retailers and eCommerce brands. Building scalable AI, real-time video, data, and commerce systems.
Full-Stack Developer
Desenvolvedor Fullstack desenvolvendo aplicações Java e Angular. Atuando com microsserviços, Kubernetes, observabilidade, segurança e resiliência de ambientes.
Tech Lead, Java
Tech Lead Java liderando arquitetura, resiliência e qualidade de soluções digitais da Go.k Digital. Integração de frontend, BFF e backend em projetos complexos e críticos.
Product Engineer, Full-Stack
Product Engineer building AI-powered web and mobile experiences at Granted, a healthcare billing platform. Shipping full-stack features that help Americans understand and reduce medical bills.
Software Engineer II – Attack Detection
Software Engineer building backend services, AI agents, and ML-driven detection workflows for Abnormal’s email security platform. Evaluating attacks, false positives, reliability, and production performance.
Full Stack Developer
Desenvolvedor Full Stack na Stefanini, ecossistema global de tecnologia e inovação. Criando e operando soluções com Java, APIs, Angular, Kubernetes e cloud.
Senior Software Development Engineer, .NET, React, Azure
Senior .NET/React/Azure engineer building distributed systems for Signify Health’s value-based healthcare platform. Developing scalable full-stack applications that transform home-based care.
Growth Engineer, Fullstack, Frontend-Focused
Growth Engineer building Vue/Nuxt acquisition experiences for Pennylane’s financial platform for French SMEs and accountants. Improving funnels, experimentation, performance, and international expansion.
Full Stack Engineer – Growth & Experimentation
Full Stack Engineer building React Native growth experiments for Kraken, a global crypto trading platform. Improving acquisition, activation, engagement, and trading revenue across Krak.
Principal Engineer – Product Engineering
Principal Engineer direcionando a evolução da plataforma de APIs, integração e IA da Sensedia. Liderando arquitetura escalável, resiliente e segura em ambiente cloud multi-tenant.
Technical Lead, QAD
Technical Lead developing and supporting QAD ERP applications for Valmont’s global infrastructure and agricultural technology business. Leading architecture, integrations, data validation, testing, and project delivery.
Full Stack Developer, Java, Angular
Desarrollador Full Stack con Java y Angular para knowmad mood, compañía de transformación digital. Construcción de soluciones backend y frontend con microservicios y tecnologías cloud.
Full Stack Developer, Java, Angular
Full Stack Developer desarrollando backend Java y frontend Angular para knowmad mood, compañía española de transformación digital. Trabajo remoto con proyectos de tecnología para clientes.
Software Technical Leader
Software Technical Leader building scalable full-stack solutions for Cisco Meraki’s networking platform. Owning architecture, reliability, and production delivery across engineering teams.
Senior Software Engineer
Senior engineer building mobile, backend, and web navigation features for Samsara’s Connected Operations Cloud. Improving routing, driver safety, and fleet visibility for physical industries.
Staff Software Engineer – Ticketing Team Lead
Staff engineer leading PHP ticketing and storage integrations for Nucleus Security’s vulnerability management platform. Building agentic automation, REST API connectors, and reliable cybersecurity workflows.
Facilitator – Engineering Software
Remote facilitator delivering PIPE-FLO, AGi32, and ElumTools training for engineering professionals. Maintaining courses, virtual classrooms, evaluations, reporting, and AI-enabled learning improvements.
Staff Software Engineer
Staff Software Engineer building dbt Fusion, a Rust-based analytics engine for Fivetran and dbt Labs. Improving query performance, correctness, orchestration, and developer experience.