Senior Data Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior Data Engineer building scalable Databricks data pipelines and enterprise integrations. Supporting Sumo Logic’s platform that unifies security and operational data for reliable digital operations.
Responsibilities:
- Design, develop, and optimize scalable data pipelines using Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake
- Design and implement Bronze, Silver, and Gold medallion architecture layers
- Build and maintain integrations with enterprise applications, REST APIs, databases, and other source systems
- Develop reusable ingestion frameworks for onboarding new data sources
- Build batch and near-real-time ingestion pipelines with incremental loads, CDC, schema evolution, and error recovery
- Configure and manage Databricks Workflows and implement pipelines using Auto Loader and Lakeflow Spark Declarative Pipelines
- Optimize Spark and Delta Lake workloads, query performance, partitioning, joins, data-skew handling, and compute costs
- Implement Unity Catalog governance, access controls, lineage, auditing, masking, and row-level security
- Maintain data workflows on Amazon Redshift and Amazon EC2
- Collaborate with Engineering and IT/Cloud teams on connectivity, IAM permissions, S3 access, deployments, and operational issues
- Follow version control, code review, automated testing, CI/CD, and technical documentation practices
- Partner with Data Analytics and business stakeholders to deliver trusted, documented, analytics-ready datasets
- Provide technical guidance and improve automation, engineering standards, platform reliability, and resource utilization
- Report directly to the Senior Manager - Business Insights & Analytics and collaborate with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders
Requirements:
- 5+ years of experience in data engineering or a related technical role, including developing and supporting production data pipelines
- Strong hands-on expertise in Databricks, including developing, troubleshooting, and optimizing production workloads
- Strong proficiency in Python/PySpark, SQL, and Apache Spark for scalable data processing and transformation
- Practical experience with Delta Lake, including table management, incremental processing, and performance optimization
- Experience with data architecture and modeling, including medallion architecture and the development of analytics-ready datasets
- Experience with Databricks Workflows or equivalent orchestration solutions
- Proven experience building data ingestion pipelines and integrating REST APIs, databases, and enterprise applications
- Understanding of reliable data processing patterns, including retries, idempotency, schema evolution, backfills, and failure recovery
- Working knowledge of AWS services and cloud infrastructure, particularly IAM roles and policies, S3 storage and access management, and secure connectivity between source systems and data platforms
- Experience with Git/GitHub, code reviews, automated testing, and CI/CD practices
- Strong problem-solving skills and the ability to independently design, build, deploy, and maintain production data solutions
- Good communication and collaboration skills to work effectively with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders
- Experience with Databricks Lakeflow Spark Declarative Pipelines, Auto Loader, and advanced incremental ingestion patterns is desirable
- Experience implementing Unity Catalog governance is desirable
- Experience with Databricks Genie, Genie Agents, and Genie Code is desirable
- Experience in AI/ML, Data Science, LLMs, contextualization, and Amazon Bedrock is desirable
Benefits:
- Remote work arrangement
- AI-powered Talent Matching tool used in recruitment
- Privacy and data protection compliance information provided
Similar Jobs
Payroll Data Migration Specialist
Payroll data migration specialist coordinating multi-country Workday migrations for Spyrosoft’s HR and payroll transformation projects. Ensuring accurate, compliant data from planning through validation.
Senior Engineer – Data Engineering, Databricks
Senior Databricks data engineer building scalable batch and streaming lakehouse pipelines. Delivering data platforms and solution accelerators for Slalom consulting clients.
Senior Data Engineer
Senior Data Engineer building AWS and Python data pipelines for Dev.Pro, a US-based software development company. Integrating POS systems with a hotel chain’s data platform and reporting solutions.
Data Engineer, Celonis
Celonis Data Engineer building SAP-connected process-mining data models for Talan’s automotive clients. Designing pipelines, OCPM event logs, and Celonis architecture for Process Intelligence initiatives.
Senior Pharmacovigilance Data Migration Consultant – Oracle Argus Safety
Pharmacovigilance data migration consultant converting PostgreSQL safety systems to Oracle Argus for life sciences clients. Supporting validation, compliance, integrations, and database modernization.
Data Engineer
Data Engineer building Snowflake, dbt, Airflow, and AWS pipelines for IDEXX’s medical data products. Enabling trusted clinical insights for analysts and veterinary health teams.
AWS Data Architect
AWS Data Architect designing S3 Data Lakes, ETL frameworks, and analytics platforms for Leighton’s cloud consultancy. Enabling governed, secure, scalable AWS data solutions.
Senior Golang Engineer, Data Processing
Senior Golang Engineer building high-throughput Go pipelines for Litmus, whose platform turns industrial machine data into usable real-time AI infrastructure. Owning streaming services across cloud and edge environments.
Senior Data Engineer
Senior Data Engineer scaling data pipelines and MLOps infrastructure for Keebo’s cloud warehouse optimization platform. Building reliable models, features, and AI-enabled engineering workflows.
Senior Data Engineer, Ads Data – AI Native
Senior Data Engineer owning Databricks pipelines, metrics, and governance for Life360’s advertising data. Improving performance, cost, reliability, and self-serve insights for teams supporting family safety products.
Senior Data Engineer
Senior Data Engineer building reliable data pipelines and models for LMI’s federal national security applications. Supporting AI, analytics, deployment, and mission-user workflows.
Staff Data Engineer, CBP
Staff Data Engineer building scalable data pipelines, models, and ontology for CBP operations. Supporting LMI’s mission-ready digital solutions in Google Cloud.
Senior Contract Software Engineer, Data & Platform
Senior contract engineer building scalable data pipelines and platform services for Nimble’s software that helps schools recruit educators. Designing reliable systems and data-intensive products for education leaders.
Senior Data Engineer, Network Platform
Senior Data Engineer building scalable, governed data platforms for Cisco’s cloud-managed networking solutions. Developing reliable pipelines, data context layers, and AI-enabled engineering workflows.
Snowflake Data Architect
Senior Snowflake Data Architect designing Snowflake healthcare data platforms for Datavail. Leading patient identity, dimensional modeling, quality controls, and engineering delivery.
Data Architect – VA OCDAO Data and Analytics
Data Architect designing enterprise data models, taxonomies, and metadata standards for the Department of Veterans Affairs. Leading business architecture content and aligning data governance structures.
Data Engineer – VA OCDAO Data and Analytics
Data Engineer building pipelines and integrations for the Department of Veterans Affairs OCDAO enterprise data catalog. Applying data quality and security controls remotely across the United States.
SQL Developer, Data Migration Specialist
SQL Developer and Data Migration Specialist configuring Momentive Software’s cloud platform for nonprofits and associations. Developing client solutions, supporting data conversion, testing deployments, and training users.
Junior Data Engineer – Snowflake, Oracle
Data Engineer Jr apoiando a migração de Data Warehouse Oracle para Snowflake na Keyrus, grupo internacional de dados e IA. Desenvolvendo pipelines, validando código e documentando processos.
Senior Data Architect – North America
Senior Data Architect shaping Snowflake architecture, financial data models, and integrations for a real estate investment firm. Improving governance, performance, quality, and reporting reliability.