Databricks Developer – AWS
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Databricks Developer designing and implementing ETL pipelines in cloud environments, focusing on data quality checks and integration with AWS.
Responsibilities:
- Design and implement scalable ETL/ELT pipelines in Databricks using PySpark/Spark SQL/Scala.
- Build and manage Delta Lake tables with ACID transactions, schema evolution, time travel, Z-Ordering, and Optimize/Vacuum routines.
- Develop batch and near-real-time pipelines (e.g., Structured Streaming, Kafka/MSK, Kinesis).
- Implement robust data quality checks (e.g., expectations/constraints, anomaly detection) and unit/integration tests.
- Integrate Databricks with Amazon S3 (bronze/silver/gold zones), AWS Glue Data Catalog or Unity Catalog, and Lake Formation where applicable.
- Configure IAM roles & instance profiles.
- Orchestrate jobs and workflows using Databricks Workflows and/or AWS Step Functions, Airflow.
- Implement CI/CD using GitHub; manage repos with Git.
- Utilize observability: CloudWatch, Databricks audit logs, metrics, cost monitors, and alerting.
Requirements:
- Bachelor’s in Computer Science, Engineering, or related field (or equivalent experience).
- 3–7 years in data engineering, including 2+ years hands-on with Databricks on AWS.
- Strong in Apache Spark (optimizations, joins, partitioning, caching)
- Solid experience with Delta Lake, S3, Glue Catalog / Unity Catalog, and Lakehouse design.
- Proficiency with SQL, performance tuning, and cost optimization on Databricks.
- Familiar with AWS services: S3, IAM, Glue, Lambda, CloudWatch, KMS, Step Functions, MSK/Kinesis, VPC.
- Version control (Git) and CI/CD for Databricks (e.g., Repos, Databricks CLI, Terraform, GitHub Actions/CodePipeline).
- Experience in Agile/Scrum delivery.
Benefits:
- Health insurance
- Flexible work arrangements
Similar Jobs
Senior Developer, Machine Identity
Senior Developer focused on access management for AI workflows at 1Password. Code, test, and maintain production software systems for new and existing product features.
Senior Manager, GRC Engineering
Senior Manager for client relationship management and compliance execution at Workstreet. Leading client engagements and developing high-performing teams in a remote-first culture.
Staff Applications Developer – Mendix
Mendix Dev Tech Lead developing and maintaining applications using Mendix platform at Dexcom. Collaborating with cross-functional teams to optimize systems and enhance user experiences.
Senior Mobile Developer
Senior Mobile Architect leading design, architecture, and delivery of mobile applications at Reply. Working fully remote from Brazil on hybrid app development projects.
Team Lead Software Development, Benefits Platform
Lead Software Engineer developing and managing a scalable Multi-Benefit platform. Seeking an experienced technical leader to drive architectural projects and build a cross-functional team.
Developer Workflow Management
Developer in Workflow Management for coeo integrating and automating workflows in IKAROS software while modeling processes and optimizing existing functionalities.
C# Developer
C# Developer responsible for developing innovative applications for coeo's web services. Work includes APIs, backend services, and optimizing database designs.
C# Developer – AI Focus
C# Developer focusing on AI to develop intelligent software solutions for data-driven processes. Collaborating closely with departments for requirements analysis and creating AI-based solutions.
Senior Product Developer
Full Stack Developer focusing on AWS product development at Gainwell. Supporting innovative technologies and solutions in health and human services.
Service Now Integration Developer – VA EHR
ServiceNow Integration Developer contributing technical expertise to large-scale health IT programs. Collaborating across teams to maintain operational continuity and support EHR integration efforts.
Developer – OutSystems
Desenvolvedor (OutSystems) na Hitss criando soluções tecnológicas e evoluindo aplicações em ambiente ágil. Participar do levantamento de requisitos e garantir a performance e escalabilidade das soluções.
Mobile Engineer
Mobile Engineer developing Android and iOS applications for a leading nearshore staff augmentation provider in Latin America. Collaborate on high-impact projects with top American companies.
Engineer / Developer – Freelance
Freelance Engineer/Developer at SPACE44 building scalable applications and cloud environments for diverse client projects. Join an expert community of engineers across various technical domains.
Flutter Developer III – Franchisee Backoffice
Flutter Developer creating high complexity mobile solutions for Grupo Boticário. Contributing to multidisciplinary teams and ensuring technical quality in digital products.
Flutter Developer – Specialist I
Flutter developer at Grupo Boticário developing mobile applications for customer management and sales. Collaborating with design and backend teams on technical solutions and improving code quality.
Mobile App Developer
Mobile App Developer building scalable iOS and Android applications for a rapidly growing AI startup. Collaborating with engineers to enhance a leading global mobile app platform.
Playable Ads Developer
Playable Ads Developer tasked with creating engaging interactive ads for Voodoo's mobile games. Collaborating with diverse teams to optimize user experience and performance.
Desenvolvedor Oracle OFS/OIC – Sênior
Develop Oracle Field Service solutions in critical operational environments, maintaining and evolving integrations and plugins. Participates in technical design, troubleshooting, and documentation processes.
Developer, XML, Data Integration
Developer working on XML-based data integrations for educational offerings at velpTEC. Focused on data structure modelling and automation for consistent information exchange.
Lead Mobile Engineer
Mobile Engineer Lead managing BeatStars’ mobile strategy while defining architecture and standards for the mobile practice. Writing production code and mentoring the mobile engineering team.