Lead Engineer – Observability Platform
Posted 3hrs ago
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Lead engineer scaling CVS Health’s observability platform for healthcare technology. Architecting OpenTelemetry pipelines for logs, metrics, and traces across cloud environments.
Responsibilities:
- Design, build, and operate core observability platform services using Go, Python, and Java (Spring Boot)
- Lead enterprise-wide adoption of OpenTelemetry, including client libraries, semantic conventions, instrumentation patterns, and Collector/agent strategy
- Architect and scale high-throughput, fault-tolerant telemetry pipelines for logs, metrics, and traces
- Develop self-service observability capabilities for onboarding, troubleshooting, and adoption
- Implement end-to-end monitoring, SLOs, health checks, and alerting for the observability platform
- Collaborate with SRE, Platform, and Cloud teams to establish reliability standards, error budgets, and incident response practices
- Participate in on-call rotations and lead incident mitigation, root-cause analysis, and post-incident reviews
- Automate operational workflows through tooling, CI/CD enhancements, and platform automation
- Secure telemetry pipelines using mTLS, secrets management, and zero-trust design patterns
- Produce and maintain technical documentation, standards, and best practices
- Gather requirements from internal engineering teams, influence roadmap prioritization, and deliver platform improvements
- Provide technical leadership through mentorship, design reviews, architectural guidance, and cross-team collaboration
Requirements:
- 7+ years of experience in Software Engineering
- 5+ years of experience with observability practices, including SLIs/SLOs/SLAs, alerting, and incident management
- 5+ years building production-grade backend services in Go and/or Java
- 5+ years implementing and operating OpenTelemetry, including OTLP, semantic conventions, and instrumentation patterns
- 5+ years with cloud-native and containerized platforms, including Docker, Kubernetes, and Argo CD
- 5+ years working with public cloud platforms such as AWS, GCP, or Azure
- 3+ years designing and scaling distributed, high-volume data pipelines
- 3+ years of experience with Helm charts and Kustomize
- 3+ years working with Grafana OSS or comparable observability backends, including Grafana, Loki, Tempo, or Mimir
- 3+ years of experience with Infrastructure as Code tools such as Terraform or CloudFormation
- 3+ years with relational databases such as PostgreSQL or MySQL
- Bachelor’s degree from an accredited university or equivalent work experience; alternatively, high school diploma plus 4 years of relevant experience
- Experience with service meshes and networking technologies such as Envoy and Istio
- Experience with commercial observability platforms such as Datadog, New Relic, or AppDynamics
- Experience with on-call scheduling tools such as OpsGenie, PagerDuty, or GoAlert
- Experience with streaming and data platforms such as Kafka or Pulsar
- Familiarity with time-series, NoSQL, or analytical databases such as ClickHouse, Bigtable, or Cassandra
- Experience with cost optimization and capacity planning for large-scale telemetry systems
- Experience with chaos engineering, resiliency testing, or fault injection
- Background in security-aware platform design, including secure service-to-service communication
- Experience mentoring senior engineers and influencing platform standards across organizations
- Strong operational experience supporting 24x7 production systems, including on-call responsibilities
- Strong technical communication and cross-team collaboration skills
- Experience operating in regulated or compliance-heavy environments such as healthcare or finance
Benefits:
- CVS Health bonus, commission or short-term incentive program in addition to the base pay range
- Medical coverage
- Dental coverage
- Vision coverage
- Paid time off
- Retirement savings options
- Wellness programs
- Other resources supporting physical, emotional, and financial well-being, based on eligibility
- Comprehensive benefits package for full-time colleagues and their families


















