Lead Databricks Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Azure Databricks Engineer building scalable data pipelines for Blend’s AI services. Optimizing Spark-based ETL/ELT, Delta Lake, and Azure analytics solutions.
Responsibilities:
- Design, develop, and maintain scalable data pipelines using Azure Databricks
- Implement ETL/ELT workflows using PySpark, Spark SQL, and Python
- Optimize Spark jobs for performance, cost, and scalability
- Work with structured and semi-structured data (Parquet, Delta, JSON, CSV)
- Build and manage Delta Lake tables, including ACID, time travel, and schema evolution
- Integrate Databricks with Azure Data Lake Storage (ADLS Gen2)
- Develop complex queries and transformations using SQL
- Collaborate with data scientists, analysts, and stakeholders to support analytics and ML use cases
- Ensure data quality, validation, and monitoring
- Follow best practices for security, access control, and governance in Azure
Requirements:
- 6+ years of experience in Data Engineering
- Strong hands-on experience with Azure Databricks
- Proficiency in Python for data processing
- Strong knowledge of SQL, including joins, window functions, and performance tuning
- Hands-on experience with Apache Spark / PySpark
- Experience working with Delta Lake
- Knowledge of Azure Data Lake Storage (ADLS Gen2)
- Understanding of distributed computing concepts
- Experience with Git version control
- Experience with Azure Data Factory
- Exposure to CI/CD pipelines, including Azure DevOps and GitHub Actions
- Basic understanding of data modeling
- Familiarity with cloud security and RBAC in Azure
- Exposure to streaming data, including Spark Structured Streaming, Event Hub, and Kafka
Similar Jobs
Lift Repair Engineer
Lift Repair Engineer repairing and maintaining lifts across Norwich for Stannah, a global access-equipment manufacturer. Diagnosing faults, replacing components, and responding to breakdowns.
Lift Repair Engineer
Lift Repair Engineer repairing and maintaining lifts for Stannah, a family-owned access-products manufacturer. Covering a structured Chichester patch across commercial, residential and public sites.
VoIP Engineer, VoIP Service Manager
VoIP Service Manager maintaining IP telephony routes for DID Global, an international business communications provider. Monitoring call quality, managing providers, and supporting sales with route pricing and client configurations.
Forward Deployed Engineer – Contract
Forward Deployed Engineer building AI-powered software for realfast, India's post-AI IT services company. Working directly with clients and founders across full-stack, data, and AI delivery.
Lift Engineer
Lift Engineer servicing and repairing lifts across Norwich for Stannah, a global mobility-equipment manufacturer. Maintaining safe operation across commercial, residential, and public sites within an allocated local route.
Ingeniero Técnico Industrial, Delineante
Ingeniero técnico industrial y delineante creando planos CAD para fabricante europeo de piezas metálicas. Preparando documentación técnica precisa para fabricación mecánica remota desde Colombia.
Ansible Engineer
Red Hat Ansible Engineer automating infrastructure operations for a United States organization. Designing playbooks, managing configurations, deployments, security tasks, and compliance reporting.
OpenShift Engineer
Red Hat OpenShift Engineer building secure, multi-tenant bare-metal clusters for an AI-focused infrastructure provider. Migrating on-premise and Azure workloads into scalable OpenShift environments.
Field Services Engineer
Field Service Engineer commissioning, maintaining, and repairing clean-energy and industrial power systems for Sensata Technologies. Troubleshooting electrical controls and training operators at U.S. customer sites with up to 50% travel.
Applied Engineer
Applied Engineer building customer integrations and accounting data pipelines for DualEntry’s AI-native ERP. Embedding on projects, shipping full-stack systems, and feeding reusable patterns into the core finance platform.
Systems Development Engineer 5 – Creative Compute, Storage
Senior systems engineer designing and automating Netflix’s globally distributed storage platforms for content creation. Owning reliability, observability, upgrades, and infrastructure-as-code delivery.
Forward Deployed Engineer
Forward Deployed Software Engineer building scalable AI underwriting SaaS features for Convr, an insurance technology company. Embedding with strategic customers while turning recurring needs into core platform capabilities.
Stairlift Engineer
Stannah Stairlift Engineer servicing and repairing domestic lifts across Birmingham for a global lift manufacturer. Managing field routes, safety checks, and on-call visits.
Jira and Confluence Data Center Engineer – ScriptRunner, Groovy, Azure
Jira-Confluence Data Center engineer rebuilding a NATO cyber operations platform mock-up. Capturing AQAP requirements and delivering video and live industry demonstrations.
Senior Supplier Development Engineer
Senior Supplier Development Engineer improving Honeywell’s industrial supplier performance and readiness. Leading supplier capability, risk, cost, quality, and OPEX improvement initiatives for Power Systems.
Senior Value Engineer – Public Sector
Senior Value Engineer driving AI-powered process transformation for public-sector organizations at Celonis, the process mining and intelligence platform leader. Owning value realization, executive workshops, transformation programs, and platform adoption.
Principal Fire Protection Engineer
Principal Fire Protection Engineer leading fire and life safety systems for SOLV Energy’s utility-scale solar, storage, and power infrastructure projects. Setting standards, overseeing consultants, and supporting commissioning.
Quantum Success Engineer
Quantum Success Engineer enabling customers on IonQ’s quantum computing platform. Supporting onboarding, troubleshooting, integrations, training, and customer-driven product improvements.
Senior Consulting Engineer, LATAM
Senior Consulting Engineer supporting ClickHouse’s real-time analytics and data warehousing platform across LATAM. Delivering customer architecture, migrations, implementations, consultative support, and high-tier troubleshooting.
Controls Engineer
Controls Engineer designing BAS/BMS and DDC systems for Ehvert’s hyperscale and AI data centers. Coordinating multidisciplinary engineering, HVAC integration, documentation, construction support, and commissioning.