Technical Lead – GPU Infrastructure
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Technical Lead owning Tether Data’s GPU compute and managed inference platform architecture. Leading distributed engineers across Slurm, Kubernetes, bare-metal GPU infrastructure, and operations.
Responsibilities:
- Own the end-to-end architecture of Tether Data's Cosmic AC GPU compute and managed inference platform
- Lead architecture proposals, high-level and low-level designs, technical reviews, and maintain the architecture baseline
- Lead and line-manage approximately twelve distributed engineers across backend, frontend, DevOps, QA, and documentation
- Establish engineering standards, conduct code and design reviews, manage release gates, hold one-to-ones, and provide growth and performance input
- Design, build, and operate a managed Slurm service for research users
- Own Slurm controller and accounting, partitions and login nodes, node onboarding, driver and CUDA baselines, stalled-job and node-health detection, drain and autohealing, storage visibility, identity, and isolation
- Own Kubernetes cluster bootstrap and lifecycle on partner-provided bare metal, NVIDIA GPU Operator and Network Operator, VM-based GPU isolation, upgrades, backup and recovery, and node replacement
- Design managed inference serving architecture, multi-GPU and multi-node parallelism, autoscaling, request routing, endpoint reliability, and confidential-compute-capable capacity
- Establish metrics, logging, alerting, and SLOs across the control plane, GPU fleet, and application tiers
- Lead incident response, post-incident reviews, and design a sustainable on-call model
- Serve as the primary technical interface to infrastructure partners and vendors
- Translate requirements into written specifications and acceptance tests, manage escalations, and contribute to capacity planning and hardware sourcing
- Work with research, model-training, and product teams to translate workloads into platform requirements and broker capacity
- Complete the platform team and set the technical bar for new engineers
Requirements:
- Eight or more years of hands-on engineering experience
- At least three years leading teams that build and operate infrastructure platforms other teams depend on
- Bachelor's or Master's degree in computer science or engineering, or equivalent practical experience
- Hands-on experience running Slurm at scale, including slurmctld, slurmdbd, partitions, QoS and priority, accounting, prolog and epilog, node health scripting, and upgrades with jobs on the system
- Experience operating an HPC or GPU training cluster for a research population is ideally preferred
- GPU fleet operation on bare metal, including NVIDIA driver and CUDA lifecycle, Fabric Manager and NVSwitch behaviour on SXM systems, DCGM-based health and utilisation, MIG, node burn-in and acceptance
- High-performance interconnect experience with InfiniBand fabric and subnet configuration, RDMA, SR-IOV, and diagnosing multi-node NCCL performance problems
- Linux systems expertise including kernel modules and drivers, PCIe passthrough and vfio-pci, cgroups and namespaces, and performance tuning for compute-heavy workloads
- Production Kubernetes operation, including control plane, upgrades, CNI and CSI, operators and custom controllers, and multi-tenancy design
- HPC storage and data movement experience with VAST, Lustre, NFS, node-local NVMe caching, and distributing large model weights and datasets across many nodes
- Observability and operations experience with Prometheus, Grafana and Loki or equivalents, SLOs, incident response, and post-incident review
- Working fluency in JavaScript and Node.js sufficient to review a control plane, CLI and worker services and make architecture decisions
- Experience shipping a platform with real users, such as a multi-tenant IaaS or PaaS or research computing service
- People management across time zones, cross-track review, written architecture decisions, and ability to communicate decisions to partners or executives
- Excellent written and spoken English
- Desirable experience with Slurm operators on Kubernetes or Kubernetes-native schedulers
- Desirable experience with modern serving stacks such as vLLM, SGLang, or TensorRT-LLM
- Desirable experience with VM and container isolation, confidential computing, Cluster API, kubeadm, Cilium, autohealing, infrastructure as code, GitOps, GPU cloud/HPC/AI lab platforms, distributed systems, and hardware-provider partnerships
Benefits:
- Fully remote work arrangement
- Occasional travel to partner sites and team events
Similar Jobs
Senior Engineering Consultant – Civil/Construction
Senior consultant leading civil construction due diligence, monitoring, and multi-site portfolios for Bureau Veritas’ commercial real estate clients. Directing multidisciplinary technical teams and client programs.
Senior Engineering Consultant – Equity, BSA
Senior engineering consultant leading PCA, Equity, and BSA due diligence projects for Bureau Veritas, a global testing, inspection, and certification company. Providing technical leadership to clients and multidisciplinary teams.
Senior Staff Software Development Engineer
Staff Applied AI Engineer building secure, production-grade AI solutions for Zscaler’s cloud security platform. Advising business leaders, shaping roadmaps, and establishing scalable engineering standards.
QA / Software Engineering Reviewer – Browser Test Validation
QA reviewer validating browser tests for Mercor’s AI-generated web application evaluation workflows. Ensuring deterministic results, reliable automation, and robust AI software benchmarks.
Staff Software Development Engineer
Staff engineer building scalable digital health platforms, microservices, and AI solutions for CVS Health. Leading architecture, delivery excellence, complex issue resolution, and developer mentorship.
Product Engineer – Core APIs, Backend
Product Engineer building scalable Golang and Node.js APIs for Trafilea’s global DTC consumer brands. Delivering AWS-powered backend systems, integrations, and AI-ready features.
Software Development Engineer III – Agency Success
Software Development Engineer III building scalable frontend, backend, billing, and onboarding systems for HighLevel’s AI-powered business operating system. Driving architecture, reliability, and performance.
Tech Lead, Senior Backend Developer – TypeScript, Node.js
Tech Lead/Senior Backend Engineer owning microservice architecture and blockchain integrations. Building stablecoin payment infrastructure for Utorg’s global fintech platform.
Lead Software Engineer
Lead software engineer coordinating software for STARK’s AI-enabled autonomous vessels and ground systems. Driving maritime product roadmaps, embedded development, simulation, and defence field testing.
Senior Software Engineer, Insurance Services
Senior software engineer developing property-attribute extraction and backend services for Porch Group's homeowners insurance platform. Remote India role using Scala, SQL, cloud infrastructure, and microservices.
Software Development Engineer III, CRM – Opportunities – AI
Backend engineer owning HighLevel’s CRM Opportunities platform. Building scalable APIs, distributed workflows, and reliable AI-assisted customer features.
Senior/Staff Java FullStack Developer
Senior Java FullStack Developer building scalable web and mobile software products at Mindera. Delivering cloud-native solutions, mentoring engineers, and shaping technical direction.
Senior Software Engineer, Full-stack Node, React
Senior full-stack engineer building TypeScript/Node.js and React systems for Airtm’s cross-border payments platform. Owning financial product features, AI-assisted development, mentoring, and production operations.
Senior Software Engineer
Senior Software Engineer building Go-based chain and provider software for Overclock Labs’ open-source Akash cloud marketplace. Debugging Kubernetes infrastructure and shipping distributed network upgrades.
Full Stack Developer
Full-stack developer expanding Magellan AI’s podcast advertising intelligence and measurement platform. Building Rails features, AI-facing APIs, and data workflows with supervised coding agents.
Software Engineer – Agentic Search Systems
Search systems engineer discussing production, relevance, retrieval, and agentic search systems with Mercor. Completing a conversational interview focused on quality, evaluation, and tradeoffs.
Software Developer
Software Developer building secure, scalable applications and APIs for CTG’s federal modernization initiatives. Collaborating across engineering, DevSecOps, infrastructure, and product teams.
Senior Software Engineer – Golang
Engenheiro Golang sênior liderando arquiteturas distribuídas, autorização de cartões e incidentes críticos. Stone, empresa brasileira de tecnologia e serviços financeiros focada em pagamentos.
Senior Full Stack Developer
Senior Full Stack Developer improving scalable reseller platforms for a global connectivity provider. Building APIs, distributed cloud systems, automation, and testing with international teams.
Software Engineer – PHP, Python
Software Engineer desarrollando aplicaciones empresariales y APIs con PHP y Python para knowmad mood, compañía de transformación digital. Trabajo remoto en proyectos internacionales de larga duración.