Data Operations Specialist
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Databricks operations specialist supporting Scicom’s production data platforms. Monitoring pipelines, resolving incidents, and improving reliability, security, and cost performance.
Responsibilities:
- Monitor the health, availability, performance, and capacity of Databricks workspaces, jobs, pipelines, clusters, SQL warehouses, and related cloud services
- Provide day-to-day operational support for production Databricks environments
- Monitor and troubleshoot ETL, ELT, streaming, API, and file-based data pipelines
- Restart, rerun, recover, reprocess, backfill, reconcile, and remediate failed workflows and data loads
- Review driver and executor logs, Spark UI metrics, cluster event logs, audit logs, and application telemetry to identify root causes
- Support Databricks Workflows, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, and batch-processing workloads
- Maintain cluster, serverless compute, instance pool, cluster policy, SQL warehouse, and environment configurations
- Coordinate platform upgrades, runtime and library updates, deployments, releases, changes, rollbacks, and post-deployment monitoring
- Serve as first- or second-level support for production incidents; triage, coordinate response, provide status updates, escalate, and conduct root-cause analyses
- Maintain incident, problem, service-request, change, deployment, and operational records
- Develop and maintain dashboards, alerts, operational reports, health indicators, SLIs, and SLOs
- Monitor job success, latency, processing duration, utilization, freshness, quality, and compute consumption
- Validate data contracts, schemas, record counts, control totals, data quality, lineage, audit evidence, and governance requirements
- Support identity, access, entitlement, Unity Catalog, secrets, security, privacy, backup, recovery, retention, archival, and disaster-recovery activities
- Analyze resource consumption, job duration, query performance, storage, and workload patterns; recommend cost and performance optimizations
- Create and maintain operational procedures, troubleshooting guides, escalation matrices, runbooks, inventories, configuration documentation, and recovery requirements
- Identify automation and continuous-improvement opportunities and participate in operational-readiness reviews
Requirements:
- Bachelor’s degree in computer science, information technology, data engineering, information systems, or a related field, or equivalent professional experience
- At least four years of experience in data operations, production support, data engineering, cloud operations, DevOps, platform support, or a related technical role
- At least two years of hands-on experience supporting Databricks environments or production Databricks workloads
- Experience supporting production ETL, ELT, batch, or streaming data pipelines
- Working knowledge of Databricks, Apache Spark, PySpark or Python, SQL, Delta Lake, Databricks Workflows or Jobs, cloud object storage, data pipeline monitoring, log analysis, and troubleshooting
- Experience investigating failed jobs, delayed pipelines, schema issues, data discrepancies, and performance problems
- Familiarity with medallion architecture and Bronze, Silver, and Gold data layers
- Experience with at least one major cloud platform: Microsoft Azure, AWS, or Google Cloud
- Experience using ticketing and service-management tools such as ServiceNow, Jira, or a comparable platform
- Experience with monitoring, alerting, logging, or observability tools
- Understanding of incident, problem, change, and release-management processes
- Ability to write and maintain technical procedures and operational documentation
- Strong analytical, organizational, communication, and troubleshooting skills
- Ability to work effectively with engineers, business analysts, architects, security teams, and nontechnical stakeholders
- Ability to support multiple production priorities in a structured and timely manner
- Preferred: Databricks Certified Data Engineer Associate or Professional; Databricks Certified Associate Developer for Apache Spark; Microsoft Azure, AWS, or Google Cloud certification
- Preferred: Experience with Unity Catalog, Delta Live Tables, Lakeflow, Auto Loader, Structured Streaming, Databricks SQL, cloud services, CI/CD tools, observability platforms, APIs, Kafka, Event Hubs, Kinesis, Airflow, dbt, Snowflake, Power BI, data contracts, schema registries, metadata management, data lineage, data-quality frameworks, OCDS, procurement, contracting, financial or public-sector data, regulated or high-security environments, SLAs and 24-hour support, disaster recovery, backup, archival, and operational-resilience testing
Similar Jobs
Operations Coordinator
Operations Coordinator managing carrier relationships, delivery scheduling, returns, billing support, and customer satisfaction for Ryder’s logistics business.
Cluster Operations & Maintenance Performance Specialist – Visayas
Remote facility maintenance specialist overseeing JLL commercial property operations across North Luzon. Improving maintenance performance, compliance, safety, and energy efficiency.
Operations and Expense Coordinator
Operations and Expense Coordinator supporting reimbursements, purchasing, travel, and reporting. Global education group empowering students through accessible, digital-first learning.
Manager, Recruiting Operations, Planning
Hilton hospitality HR manager leading recruiting onboarding and analytics operations. Improving candidate experience, reporting, workforce planning, and service delivery through data-driven process optimization.
Senior Roaming Operations Engineer
Senior Roaming Operations Engineer supporting BICS’s global telecom connectivity network. Managing roaming services, signaling operations, incidents, maintenance, and international customer support.
Senior Roaming Operations Engineer
Senior Roaming Operations Engineer supporting BICS’s global roaming and connectivity network. Managing second-line operations, incidents, service quality, and telecom network rollouts.
Junior Operations Analyst
Analista de Operações Júnior validando pedidos, prescrições e documentos clínicos. Encaminhamento e acompanhamento de pedidos para ampliar o acesso sustentável a medicamentos.
Head of Office of Operations
Head of Operations partnering with OCHIN’s COO to drive enterprise execution, performance management, and transformation. Advancing scalable healthcare operations for underserved communities.
Operations System Analyst
Operations Systems Analyst improving Scan.com’s national diagnostic imaging network through CRM, reporting, workflow automation, and AI-powered operational systems.
Media Operations Analyst
Marketing Operations Analyst scaling Forbes Advisor’s emerging OpenAI Ads channel. Operating campaigns, experiments, measurement, and reporting for consumer advice and product-comparison content.
Operations Associate
Operations Associate managing recruiting workflows, candidate communications, data accuracy, and reporting. Supporting Rockstar’s GenAI-powered recruiting infrastructure for U.S.-based clients.
Head of Operations and Strategy
Head of Strategy & Operations scaling Nirvana’s AI-driven commercial insurance platform. Driving planning, operations, automation, marketing, safety, and executive initiatives.
Operations Manager – Pod Lead
Operations Manager leading remote pods of Amazon Account Specialists for RocketAMS, an Australian e-commerce agency. Coaching teams, ensuring quality, managing workloads, and supporting client accounts.
Operations Manager, Airbnb
Gestionnaire indépendant optimisant des locations Airbnb, Booking et Abritel pour ALB Conciergerie. Développement d'un portefeuille de propriétaires et coordination des séjours en France.
Operations Intern
Operations intern assessing student business ideas and school progress for Udhyam Learning Foundation’s entrepreneurship education programs. Reviewing submissions, tracking quality, and identifying implementation improvements.
Regional Operations Manager – Detainee Transport Operations
Regional operations manager overseeing GardaWorld Federal Services’ government-contract detainee transportation across Texas. Leading dispersed teams, staffing, compliance, safety, security, and field operations.
Fleet Manager – Detainee Transport Operations
Fleet Manager overseeing government-contracted detainee-transport vehicles across Texas. Managing maintenance, compliance, vendors, deployments, and operational readiness with 75% travel.
Operations Manager, Detainee Transport Operations
Operations Manager directing secure detainee transport, dispatch, fleet maintenance, and driver teams. GardaWorld Federal Services supports federal security and transportation operations across Texas and neighboring states.
Quality Control Lead – Detainee Transport Operations
Quality Control Lead auditing detainee transport operations for GardaWorld Federal Services across Texas. Enforcing federal, PBNDS, DOT, and contractual compliance through inspections and corrective actions.
Admin Manager – Detainee Transport Operations
Admin Manager overseeing records, staffing, compliance, and reporting for GardaWorld Federal Services’ government-contracted detainee transportation operations across Texas. Remote statewide role with up to 50% travel.