Senior Staff Engineer – Infrastructure Systems
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior Staff Engineer owning GPU drivers, CUDA, Kubernetes, and diagnostics for Runware's AI supercomputer. Building reliable GPU infrastructure, automated fault recovery, and fleet-wide software baselines.
Responsibilities:
- Own the GPU software baseline across kernel, NVIDIA driver, firmware, CUDA runtime and libraries, and container toolkit
- Qualify driver and CUDA releases across GPU generations, frameworks, and model families
- Maintain compatibility matrices and identify correctness, performance, and memory regressions
- Root-cause GPU failures including Xid errors, CUDA errors, cuBLAS failures, out-of-memory issues, fragmentation, and GPUs falling off the bus
- Build GPU acceptance, burn-in, diagnostics, memory, interconnect, and real-workload tests
- Make GPU health visible and self-healing using telemetry from DCGM, drivers, kernel logs, and BMC
- Automate GPU fault classification, node draining, resets, and replacement workflows
- Own GPU workload exposure through container runtime, device plugins, GPU Operator, MIG, sharing policies, and tenant isolation
- Bring up new GPU generations and pods with hardware and node operations teams
- Validate drivers, firmware, NVLink, and NVSwitch and automate future bring-up
- Benchmark image, video, audio, and LLM inference across GPU generations and software stacks
- Feed performance results into placement, pricing, and hardware purchasing decisions
- Work with NVIDIA and upstream projects on reproducible driver and firmware bugs and open-source fixes
- Use AI agents for log and Xid triage, test generation, release qualification, and runbook automation
- Collaborate with node operations, Managed Kubernetes, inference performance, and Sonic Inference Engine teams
Requirements:
- 12+ years building and operating systems software, including several years with GPU software running in production at scale
- Deep hands-on knowledge of the NVIDIA software stack: kernel driver and modules, CUDA runtime and libraries, NVML, DCGM, Container Toolkit, and version interactions
- Experience root-causing GPU failures across hardware, firmware, driver, runtime, and application layers
- Knowledge of Xid errors, ECC and row remapping, PCIe and NVLink errors, thermal faults, and memory faults
- Strong Linux internals knowledge: kernel, device drivers and modules, PCIe, NUMA, IOMMU, cgroups, containers, perf, eBPF, and ftrace
- Daily coding experience
- Fluent in Python and at least one systems language: C, C++, Rust, or Go
- Comfortable using CUDA to write tests or reproduce bugs
- Experience running GPUs under Kubernetes and on bare metal, including device plugins, GPU Operator, MIG, image builds, and large-scale provisioning
- Working knowledge of GPU inference frameworks including PyTorch, TensorRT, and ONNX Runtime
- Knowledge of memory allocation, compilation and caches, and multi-GPU execution
- Clear technical writing for design documents, postmortems, and ticket updates
- Daily use of AI tools to accelerate output
- Comfortable owning outcomes in a fast-moving, ambiguous environment
- Nice to have: GPU driver, CUDA library, or GPU system software experience at a GPU vendor, cloud provider, or HPC centre
- Nice to have: Blackwell hardware, NVLink/NVSwitch topologies, and liquid-cooled racks
- Nice to have: NCCL, GPUDirect, RDMA over RoCE or InfiniBand
- Nice to have: Firmware qualification across BIOS, BMC, NIC, and GPU
- Nice to have: Inference serving internals including TensorRT-LLM, vLLM, SGLang, or custom diffusion/video pipelines
- Nice to have: AMD or custom accelerator software stacks
- Nice to have: An open-source project used in production
- Nice to have: Multi-tenant isolation and confidential computing on shared GPU hardware
- Nice to have: Experience running a 24x7 service with published service levels
Benefits:
- Remote-first work arrangement
- In-person company retreats twice a year
- Core collaboration hours with flexible hours outside them
- Flexible schedule
- Downtime after major release pushes
- Generous paid time off: vacation, sick days, public holidays
- Meaningful stock options
- Paid maternity, paternity, and caregiver leave
Similar Jobs
Principal Infrastructure Architect
Principal Infrastructure Architect designing cloud and on-premise infrastructure for Cincom’s enterprise software products. Owning sizing, performance testing, Kubernetes strategy and hosted-service cost models.
IT Infrastructure Operations Engineer I
Entry-level IT infrastructure engineer supporting servers, networks, and service tickets for a global 24x7 operations team in India. Troubleshooting hardware, monitoring systems, and escalating incidents.
Senior Infrastructure Engineer
Senior Infrastructure Engineer building Walrus’s B2B payment verification platform. Designing and owning payments, reputation, and authentication modules.
Cloud Infrastructure Engineer
Cloud Infrastructure Engineer building secure, scalable, automated infrastructure for Bright Vision Technologies, a U.S. cloud, AI, data, and enterprise solutions company.
Senior Payments Infrastructure Engineer – Crypto, Wallets & Blockchain
Senior engineer completing secure crypto payments, wallets, ledgers, and blockchain infrastructure for OleOle’s global football fan platform. Deploying and operating production-ready financial systems.
OpenStack Infrastructure Engineer
OpenStack Infrastructure Engineer building and scaling xneelo’s reliable web hosting compute and storage infrastructure. Operating OpenStack, Ceph and Linux systems in a distributed team.
Maximo Infrastructure Architect – Engineer
MaxAccelerate Maximo architect delivering end-to-end MAS 9 infrastructure for global enterprise clients. Building OpenShift, cloud, database, automation and AI-enabled asset management platforms.
Senior Cloud Infrastructure Engineer
Senior Cloud-Infrastructure Engineer building AWS-native CI/CD platforms for Finanzen.net’s investment and financial-information services. Improving developer tooling and feature-team delivery.
Data Center Infrastructure Architect
Data center architect designing secure, resilient infrastructure across networking, compute, storage, virtualization, and security. Guiding enterprise implementations for ERP Suites’ 50+ customers.
Senior Frontend Infrastructure Engineer
Senior Frontend Infrastructure Engineer evolving Seeq’s React infrastructure, design systems, and reusable patterns. Building frontend foundations for Seeq’s industrial process-data insight software.
Senior Frontend Infrastructure Engineer
Senior Frontend Infrastructure Engineer evolving React infrastructure, design systems, and reusable patterns. Building foundations for Seeq’s industrial process data insight software.
Senior Director, Infrastructure Engineering
Senior Director leading multidisciplinary engineering for Volta’s vertically integrated AI infrastructure platform. Developing standards and reference designs across global AI factories.
Senior Infrastructure Engineer
Senior Infrastructure Engineer operating d-Matrix’s AI infrastructure, Kubernetes clusters, and CI/CD systems. Resolving incidents, automating recurring fixes, and supporting inference environments.
Senior Infrastructure Operations Engineer
Senior Infrastructure Operations Engineer running Linux, Hetzner, and infrastructure-as-code systems. Improving reliability and automation for IP Fabric’s network-automation platform.
Senior DevOps, Infrastructure Engineer
Senior DevOps and Infrastructure Engineer leading teams and reliability for Smartcat’s AI-powered multilingual content platform. Improving CI/CD, cloud scalability, and production availability.
Cloud Infrastructure Architect
Cloud Infrastructure Architect modernizing Azure platforms for Preferred Mutual Insurance. Leading cloud migration, infrastructure automation, DevSecOps, security, and AI-driven operational transformation.
Infrastructure Engineer, Db2
Infrastructure Engineer managing Db2 for z/OS infrastructure at a UK bank. Automating infrastructure, CI/CD, observability, and database operations.
Security & Infrastructure Engineer
Security & Infrastructure Engineer securing Janus’s healthcare process-improvement platform. Owning penetration testing, SOC 2 compliance, Microsoft 365 identity, AWS security, and DevOps infrastructure protection.
Senior Infrastructure Engineer – Citrix VDI
Senior Infrastructure Engineer owning Citrix and other VDI platforms for IT services clients across LATAM. Managing image lifecycles, incidents, automation, and enterprise VDI projects.
Senior Infrastructure Engineer – Virtualization
Senior Infrastructure Engineer operating VMware Cloud Foundation, NSX, and vSphere platforms for ITTConnect’s managed-services client. Leading virtualization operations, incidents, automation, upgrades, and migrations remotely across LATAM.