Senior Site Reliability Engineer

Posted 1hrs ago

Employment Information

Education
Salary
Experience
Job Type

Report this job

Job expired or something wrong with this job?

Job Description

Senior SRE improving Java application reliability for Hard Rock Digital’s online sportsbook, casino, and social gaming platform. Building LLM agents and observability automation to reduce incidents and operational toil.

Responsibilities:

  • Ensure availability, reliability, and performance of high-traffic Java-based applications in distributed environments
  • Troubleshoot and resolve complex issues across production and non-production environments
  • Participate in pre- and post-deployment performance testing and monitoring
  • Optimize Java application performance through JVM tuning, efficient resource utilization, and horizontal scaling
  • Deploy and manage Grafana, Prometheus, Loki, Mimir, and Alloy for monitoring, logging, and alerting
  • Implement observability strategies and maintain dashboards, alerts, and log queries
  • Integrate AI/ML models for anomaly detection, predictive alerting, intelligent alert correlation, and noise reduction
  • Design, build, and operate agentic AI workflows for alert triage, root cause analysis, runbook execution, and incident summarization
  • Develop tool-calling LLM agents interacting with Kubernetes, Grafana, Jira, Slack, and PagerDuty APIs
  • Build and maintain MCP servers and integrations exposing internal systems to AI agents
  • Evaluate and operationalize LLM frameworks and orchestration platforms
  • Implement guardrails, evaluation harnesses, and feedback loops for safe and accurate AI agents
  • Champion AI-assisted development and operations practices
  • Support incident response, conduct post-mortems, and identify root causes
  • Use AI tools to accelerate incident timelines and generate post-mortem drafts
  • Document and share lessons learned
  • Identify repetitive workflows and build AI-augmented or automated replacements
  • Build self-service tools and chatbot interfaces for system status, log retrieval, and SOP execution
  • Measure and report toil-reduction metrics
  • Collaborate with developers, architects, data/ML engineers, DevOps, and NOC teams
  • Communicate SRE practices, AI/automation capabilities, and operational insights to technical and non-technical stakeholders
  • Provide feedback on application performance and observability metrics

Requirements:

  • Degree in Computer Science or a related field, or equivalent professional experience
  • 5+ years in SRE, DevOps, or similar infrastructure roles with experience managing large-scale, high-availability production systems
  • 3+ years hands-on experience managing production Kubernetes clusters, including architecture, networking, storage, and security
  • Experience with cluster autoscaling (Karpenter), upgrades, and multi-cluster management
  • Proficiency with kubectl, Helm, Kubernetes operators, and container orchestration troubleshooting
  • Advanced expertise with the Grafana observability stack, including dashboards, alerting, visualization, and Grafana Alloy
  • Proficiency in PromQL and experience with Loki for log aggregation and analysis
  • Hands-on experience managing Java-based applications in distributed environments, including JVM tuning and optimization
  • Cloud platform expertise; AWS preferred, with GCP or Azure also valued
  • Familiarity with Terraform/Terragrunt or Ansible
  • ArgoCD proficiency for GitOps workflows and continuous deployment
  • Strong scripting abilities in Python, Bash, or Go, with experience building CI/CD pipelines and deployment automation
  • Proven track record with on-call rotations, incident response, and root cause analysis
  • 1+ years of practical experience building or operating AI/LLM-powered tools, agents, or workflows in production or production-adjacent contexts
  • Ability to design agentic systems using tool calling, RAG, or multi-step reasoning
  • Experience integrating LLM APIs such as Anthropic Claude, OpenAI, or open-source models into backend services or automation pipelines
  • Familiarity with at least one agentic orchestration framework or workflow engine, such as LangChain, LangGraph, CrewAI, n8n, Temporal, or equivalent
  • Understanding of prompt engineering best practices, including structured outputs, system prompts, and few-shot examples
  • Familiarity with AI-assisted coding tools such as Claude Code, Codex, and Cursor
  • Experience building or consuming MCP servers
  • Awareness of AI safety, hallucination mitigation, and human-in-the-loop design patterns
  • Strong written and verbal communication skills
  • Ability to translate complex AI and infrastructure concepts for diverse audiences
  • Ability to mentor junior team members on traditional SRE practices and emerging AI-driven approaches
  • Positive attitude and openness to constructive feedback

Benefits:

  • Autonomy to experiment with cutting-edge AI tools
  • Leadership backing to deploy AI tools in production
  • Mandate to measurably reduce operational toil through intelligent systems