Staff Software Engineer – Cloud Platform, Kafka
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Staff Cloud Platform Engineer facilitating cloud-based Kafka deployment on GCP and AWS. Collaborating with teams to enhance event streaming infrastructure at a leading CSP.
Responsibilities:
- Design, provision, and manage Apache Kafka clusters (self-managed on GCP/AWS or via Confluent Platform / MSK).
- Configure and tune brokers, ZooKeeper/KRaft, topics, partitions, replication factors, and retention policies for high throughput and low latency.
- Perform cluster upgrades, rolling restarts, and broker replacements with zero downtime.
- Implement and manage Kafka Connect pipelines for data ingestion and egress across heterogeneous systems.
- Administer Kafka Streams and ksqlDB deployments for real-time stream processing workloads.
- Maintain Schema Registry and enforce schema governance standards across teams.
- Define and track SLIs/SLOs for consumer lag, throughput, end-to-end latency, and broker health.
- Design and implement cloud infrastructure using IaC – Terraform
- Build automated deployment pipelines for Kafka configuration changes using GitOps workflows (ArgoCD, Flux).
- Create self-service tooling and runbooks to reduce toil for development teams.
- Automate topic provisioning, ACL management, and schema registration via APIs and CLI tooling.
- Integrate tools like GitLab CI/CD , or Cloud Build for automated testing and deployment.
- Ensure seamless integration of data pipelines with other GCP services like Big Query, Cloud Storage.
- Monitor and Optimize performance, reliability, and cost of Kafka and streaming pipelines.
- Implement security best practices for GCP resources, including IAM policies, encryption, and network security.
- Ensure Observability is an integral part of the infrastructure platforms and provides adequate visibility about their health, utilization, and cost.
- Collaborate extensively with cross functional teams to understand their requirements; educate them through documentation/trainings and improve the adoption of the platforms/tools.
Requirements:
- 10+ years of overall experience in DevOps cloud engineering, or data engineering.
- 5+ years of experience in Kafka at production scale.
- Deep expertise in Kafka internals: replication protocol, log compaction, consumer group coordination, partition leadership, and KRaft mode
- Proficiency with container orchestration (Kubernetes / Helm) and deploying Kafka via Strimzi, Confluent Operator, or equivalent
- Strong understanding of networking (VPC, peering, private endpoints, DNS, load balancing) in cloud environments.
- Hands-on experience with Kafka Connect, Schema Registry, and at least one stream processing framework (Kafka Streams, Flink, Spark Structured Streaming).
- Proficiency in Google Cloud Platform (GCP) services, including Dataflow, Pub/Sub, Kafka, Dataproc, Big Query, and Cloud Storage.
- Expertise in Infrastructure as Code (IaC) tools like Terraform or Cloud Deployment Manager.
- Familiarity with data orchestration tools like Apache Airflow or Cloud Composer.
- Experience with CI/CD tools like Jenkins, GitLab CI/CD, or Cloud Build.
- Knowledge of containerization and orchestration tools like Docker and Kubernetes.
- Strong scripting skills for automation (e.g., Bash, Python).
- Experience with monitoring tools like Cloud Monitoring, Prometheus, and Grafana.
- Familiarity with logging tools like Cloud Logging or ELK Stack.
- Strong problem-solving and analytical skills.
- Excellent communication and collaboration abilities.
- Ability to work in a fast-paced, agile environment.
Benefits:
- Health insurance
- 401(k) matching
- Flexible work hours
- Paid time off
Similar Jobs
Senior Full Stack Engineer
Designs and develops scalable web applications across front-end and back-end as a Senior Full Stack Engineer at ARC-One. Collaborates in a regulated SaaS environment with a focus on customer needs and continuous improvement.
Software Engineer III
Software Engineer III developing scalable cloud and AI platforms for CData Software. Contributing to multi-tenant architecture and collaborating with cross-functional teams to solve complex problems.
Full Stack Developer, SAP Commerce Cloud
Experienced SAP Commerce Cloud Full Stack Developer developing and implementing tech solutions across the full stack. Collaborating with teams using Angular and Spartacus in a remote environment.
Manager – Software Engineering
Manager - Software Engineering managing backend teams with a focus on Ruby on Rails. Driving scalable architectures and ensuring reliable software delivery in Pune, India.
Full Stack Engineer
Full Stack Engineer developing and implementing applications in alignment with EDC’s AI & Digital Transformation strategy. Collaborating cross-functionally to drive high-priority business outcomes.
Senior Full Stack Engineer
Lead full-stack applications aligned with EDC’s AI & Digital Transformation strategy. Design solutions and mentor teams while collaborating cross-functionally across organizations.
Full Stack Developer – 6-month contract
Full Stack Developer for Signal49 Research, creating interactive dashboards and reporting tools. Work collaboratively with internal clients and data teams in a remote setting.
Software Architect
Software Architect responsible for designing scalable and resilient systems at DraftKings. Collaborating with cross-functional teams to shape architectural decisions and improve system performance.
Software Engineer, AI
Software Engineer developing AI solutions and optimizing RAG pipelines for engineering teams at Violet Labs. Collaborating with various stakeholders to improve application performance and data quality.
Software Engineer
Founding CTO at Omnia to build an AI platform redefining enterprise human capital. Leading technical vision and mentoring a high-performing engineering team in Switzerland.
Senior Software Engineer
Senior Software Engineer developing scalable backend systems using Node and Python. Collaborating with engineers and designers to enhance user experience for small business customers.
Staff Software Engineer – API Testing, Automation
Staff Engineer leading testing and automation for Zscaler's security solutions. Collaborating with teams to enhance cloud-native quality and operational efficiency.
Senior Software Developer, Facility Management
Software Developer managing facility management applications development for public sector. Expanding product portfolio with cloud and on-premises solutions and integrating multiple systems and applications.
Senior Software Developer, Facility Management
Senior Software Developer in a growing team enhancing facility management products for government sectors, both on cloud and on-premise. Agile methodology with modern development tools.
Fullstack Engineer
Fullstack Engineers with ambition to prove technical proficiency and compete with U.S. talent. Join Silver.dev for potential U.S. immigration sponsorship through exceptional skill validation.
Associate Staff Engineer, SAP PS
Associate Staff Engineer, SAP PS at Nagarro focusing on SAP S/4HANA and project creation. Collaborating on functional requirements and leading application development activities with a global team.
Senior Staff Engineer, SAP Spend Management
Senior Staff Engineer for SAP-Spend-Management at Nagarro, a Digital Product Engineering company growing globally.
Associate Principal Engineer, SAP Spend Management – Fieldglass
Associate Principal Engineer focusing on SAP Spend Management, leading supplier enablement and VMS implementations at Nagarro.
Principal Engineer, SAP Spend Management - Fieldglass
Principal Engineer responsible for SAP-Spend-Management, managing supplier enablement and procurement transformation at Nagarro's global teams.
Associate Staff Engineer, SAP Treasury
Associate Staff Engineer in SAP Treasury at Nagarro, focusing on SAP FI modules and financial systems solutions. Working in a dynamic culture to analyze and meet client needs with innovative solutions.