Staff Observability Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Staff Observability Engineer designing AI observability and reliability systems for Tealium’s customer data platform. Building telemetry, SLOs, tracing, and cost controls across enterprise services.
Responsibilities:
- Lead end-to-end observability design, telemetry schemas, and OpenTelemetry pipeline architectures
- Partner with engineering and product teams to define SLIs, SLOs, error budgets, and production-readiness requirements
- Architect AI observability solutions for Amazon Bedrock and agentic workflows
- Track model selection, latency, retries, token usage, tool execution, and cost attribution
- Establish traceability across user requests, prompts, model calls, retrieved context, downstream services, and final responses
- Maintain data privacy and security controls
- Develop dashboards, alerts, SLOs, runbooks, and operational views
- Drive incident diagnosis, performance optimization, and economic sustainability
- Define measurable signals for response quality, groundedness, guardrail outcomes, and task completion
- Participate in an on-call rotation approximately 20% of the time
- Drive failure injection, production testing, and capacity planning
- Collaborate with SRE, MLOps, data engineering, security, product, and non-technical stakeholders
Requirements:
- 6+ years in Site Reliability, Observability, or Platform Engineering supporting 24x7x365 production systems
- Deep experience with OpenTelemetry and platforms such as Datadog, Sumo Logic, Prometheus, or Grafana
- Experience designing observability for distributed systems, APIs, event-driven architectures, and asynchronous workflows
- Hands-on experience instrumenting AI/ML or GenAI systems
- Proficiency with Amazon Bedrock or similar platforms
- Familiarity with agentic workflows, prompt engineering, vector databases, RAG architectures, LangChain, LlamaIndex, or SageMaker
- Proficiency in Java, Python, or Go
- Strong AWS expertise, including cloud networking, IAM, security, and service quotas
- Experience with IaC, CI/CD, and containers, including Terraform, Kubernetes, Argo CD, Jenkins, or GitHub Actions
- Experience with data modeling, telemetry pipelines, cardinality management, retention, responsible AI, and data privacy controls
- Strong communication, mentoring, and cross-functional leadership skills
Benefits:
- Performance-based bonus eligibility
- Equity options
- 15 hours of paid work time annually for volunteer activities and programs
- Remote-first working
- New hire stipends for home office setup
- New hire equity grants
- Paid time off
- Extended paid parental leave
- Company holidays
- Health and wellness programs covering physical, mental, social, and financial well-being
- Professional development opportunities
- Over 6,000 on-demand courses
- Manager and leadership development programs
- Health and related benefits programs
Similar Jobs
Remediation Engineer
Remediation Engineer hardening infrastructure, managing Microsoft and Azure environments, and remediating vulnerabilities for CompassMSP managed-services clients.
Forward Deployed Engineer
Forward Deployed Engineer supporting goTenna’s off-grid mobile mesh networking products. Providing technical support, customer training, installations, and presales assistance for public safety and military customers.
Junior Technical Delivery Engineer
Junior Technical Delivery Engineer deploying Capacity/Verbio AI support automation solutions in customer environments. Troubleshooting Linux-based deployments and supporting customers, project managers, and senior engineers.
Industrial Electrical Maintenance Engineer
Engenheiro de manutenção elétrica industrial projetando sistemas de energia, UPS e contingência para ambientes de TI. Atuando com normas, comissionamento e infraestrutura tecnológica na Stefanini.
M365, Identity Engineer
M365 and identity engineer designing and securing Microsoft 365, Entra ID, Okta, and hybrid identity environments for Amentum’s engineering and technology operations.
Senior Azure IaC Engineer
Senior Azure Engineer building and governing Azure infrastructure for AnaVation’s U.S. Federal Government clients. Automating deployments and supporting a digital evidence platform.
Junior Azure IaC Engineer
Azure IaC Engineer building cloud infrastructure and automation for AnaVation’s federal intelligence customers. Supporting Azure services, DevOps practices, and enterprise digital evidence platforms.
Managing Engineer, C#/ASP.NET
Managing Engineer leading C#/ASP.NET teams at Allstate, an insurance and protection company. Building enterprise applications while guiding architecture, delivery, and engineering development.
Firmware Development Engineer Co-op
Firmware engineering co-op developing microcode, firmware, FPGA, and DSP interfaces for Intel custom silicon. Testing implementations and collaborating with experienced engineers remotely across Canada.
Lift Repair Engineer
Lift Repair Engineer repairing and fault-finding lifts across Basingstoke for Stannah, a global access-equipment manufacturer. Diagnosing electrical and mechanical faults and restoring equipment safely.
Claims Desktop Motor Engineer
Claims Desktop Motor Engineer assessing vehicle repairs, total losses, salvage, and repair costs for Acorn Insurance. Managing motor claims outcomes and repairer relationships remotely across the UK.
Senior ERP Engineer
Senior ERP Engineer managing Deltek Vantagepoint and Trimble Spectrum for an engineering and mechanical services organization. Owning administration, reporting, integrations, upgrades, security, and audit support.
Engineer Manager – Network Solutions
Engineering Manager owning Nexcess Network Solutions architecture, roadmap, and delivery. Leading AI-assisted engineering for cloud networking, load balancing, firewalls, VPN, and VPC products.
Virtualization Engineer
Virtualization Engineer operating enterprise VMware, networking, and storage platforms. Supporting hybrid cloud environments for a U.S. technology consulting and software development company.
Senior DNS Engineer
Senior DNS Engineer building scalable DNS control-plane software for DigitalOcean's cloud platform. Developing APIs, improving reliability, and supporting high-performance networking products.
Senior DNS Engineer
Senior DNS Engineer building scalable control-plane software for DigitalOcean’s cloud platform. Developing APIs, improving DNS products, and operating high-performance networking systems.
Senior DNS Engineer
Senior DNS Engineer building scalable APIs and control-plane software for DigitalOcean’s cloud platform. Improving DNS product reliability, performance, tooling, and customer experience.
Senior DNS Engineer
Senior DNS Engineer building scalable DNS control-plane software for DigitalOcean’s cloud platform. Developing APIs, improving reliability, and collaborating across Networking teams.
Senior Forward Deployed Engineer
Senior Forward Deployed Engineer deploying Synapse Analytics products across banking and enterprise environments. Designing integrations, infrastructure, and production solutions for restricted client systems.
ROS Engineer
ROS Engineer migrating Spyrosoft’s autonomous UGV platform from ROS 1 to ROS 2. Integrating navigation, localization, vehicle control, sensors, and CAN-connected hardware.