Senior Databricks Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior Databricks Engineer leading architecture and optimization of next-generation Lakehouse platform. Drive technical direction and deliver scalable data solutions in remote work setup.
Responsibilities:
- Ingestion & Transformation: Design and optimize high-volume ETL/ELT pipelines using Delta Live Tables (DLT) and PySpark, ensuring data integrity across the Bronze, Silver, and Gold layers.
- Workflow Orchestration: Develop and maintain sophisticated pipelines using Databricks Workflows or Airflow, focusing on modularity, reusability, and automated error handling.
- Streaming & Real-time Integration: Implement real-time data flows utilizing Structured Streaming and Kafka/Event Hubs to enable immediate data availability for downstream consumption.
- Data Security & Privacy: Enforce data anonymization and fine-grained access controls to ensure compliance with global regulations (GDPR/CCPA/HIPAA).
- DataOps & DevOps: Implement CI/CD patterns using Databricks Asset Bundles (DABs), Terraform, and Git to automate environment parity and deployments.
- Open Table Formats: Manage and optimize Delta Lake storage, utilizing advanced features like Liquid Clustering, Z-Ordering, and Change Data Feed (CDF).
- Compute Engine Optimization: Drive cost efficiency and performance by optimizing Spark configurations, Photon engine utilization, and Serverless SQL Warehouses.
- Observability & Monitoring: Integrate comprehensive monitoring and alerting (e.g., Databricks System Tables, Grafana, or Splunk) to rapidly identify bottlenecks and troubleshoot production issues.
Requirements:
- 6+ Years of hands-on, progressive experience in Data Engineering, with at least 5 years focused heavily on the Databricks platform.
- Architectural Understanding: Expert knowledge of Medallion Architecture, Data Vault 2.0 or Dimensional Modeling, and modern Lakehouse design patterns.
- Scale Expertise: Proven track record of building and managing large-scale data infrastructure (Petabyte-scale) in cloud-native environments.
- Industry Experience: Experience in the Insurance or Financial Services industry is preferred (focusing on claims, policy, or risk data).
- Technical Toolset:
- Cloud Environment: Azure (preferred), AWS.
- Databricks Stack: Unity Catalog, Delta Live Tables, Databricks SQL, MLflow.
- Core Languages: Expert-level SQL, Python, and PySpark.
- Supporting Tools: dbt (Databricks adapter), Git, and Orchestration tools.
Benefits:
- Work From Home
Similar Jobs
Senior Mainframe Automation, Migration Engineer
Senior Mainframe Automation Migration Engineer leading mainframe automation migrations for Phizenix. Engage in configuration, development, and collaboration with operations for enhanced automation.
T300 Stress Calculation Engineer
Ingénieur Calcul T300 Stress responsible for analyzing aircraft structure and stress impact. Collaborating with design teams to ensure quality deliverables and managing technical support.
Finite Element Analysis (FEA) Engineer
Technical role in Expleo focused on FEM calculations for aerospace projects. Involves coaching engineers and collaborating with clients on methodology and processes.
Static Calculation Engineer – Aerospace
Aéronautique static calculation engineer developing and certifying aircraft designs during maturity phases. Collaborating on large projects in Toulouse.
Ingeniero/a de Automatización, Junior
Junior Automation Engineer working remotely on an international banking project. Requiring CI/CD tools and cloud knowledge with a good level of English.
Engineer II
Engineer II role at biotech company focusing on electromechanical device development and lifecycle management. Responsibilities include ensuring product safety, performance, and collaboration with multi-functional teams.
Senior Project Engineer – Nuclear Operational Readiness
Senior Project Engineer directing nuclear project engineering and design at GE Vernova. Responsible for managing technical interfaces and project delivery in a complex environment.
Pulp & Paper Process Engineer II
Process Engineer designing complete process engineering assignments for chemical and petroleum projects. Responsible for safety, project criteria, and mentoring within a team environment.
Senior Vendavo Configuration Engineer
Senior Vendavo Configuration Engineer leading configuration and optimization of Vendavo solutions for enterprise clients. Collaborating with pricing analysts and stakeholders to improve margins in a remote role.
Unmanned Systems Chief Engineer
Chief Engineer for Unmanned Systems at CACI, focusing on strategy development and execution for intel related engineering processes. Collaborating with various stakeholders to ensure project success.
OSP Engineer II
OSP Engineer responsible for designing and maintaining telecommunications infrastructure for Pearce Services. Collaborating with various stakeholders to ensure project efficiency and compliance.
Forward Deployed Engineer
Forward Deployed Engineer for ModSquad, designing, building, and deploying complex AI systems for high-stakes environments. Involves collaborative engagements with clients and deep system integration challenges.
Engineer – Identity Solutions
Engineer at DGR Systems focusing on Identity and Access Management solutions. Delivering projects to enhance security and operational efficiency in customer environments.
Ingeniero/a ILS Naval – Aprovisionamiento
Ingeniero/a ILS Naval ensuring availability of spare parts for military ships. Working in a multinational consulting firm specialized in engineering services.
Ingeniero/a ILS Naval – Control de Configuración
Engineer joining multinational consultancy ATEXIS, specialized in naval defense configuration control. Responsible for logistics configuration integrity of military vessels throughout their lifecycle.
Ingeniero ILS Naval, Mantenimiento
Ingeniero/a ILS Naval responsible for maintenance strategies of military vessels. Join ATEXIS, an engineering consultancy specializing in aerospace, defense, and maritime services.
Power Systems Advisory Engineer
Power Systems Advisory Engineer supporting smart grid technology for electric cooperatives. Leading engineering insights and project execution in a collaborative environment.
Senior Electrical Commissioning Engineer
Senior Electrical Commissioning Engineer for data center projects at JLL. Responsible for commissioning, QA/QC processes, and team management.
Process Safety Engineer
Process Safety Engineer developing and implementing process safety programs for safe manufacturing at Solenis. Providing technical leadership in hazard analysis, risk management, and compliance auditing.
Distinguished Engineer
Distinguished Engineer at Home Depot shaping technology strategies and mentoring engineering talent. Focus on Marketing Tech, AI strategies, and enterprise architecture.