Senior Data Engineer, Databricks
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Engenheiro de Dados Sênior atuando em migração e engenharia de dados no Azure Databricks. Automatizando pipelines e assegurando governança, desempenho e integridade em ambientes críticos.
Responsibilities:
- Migrate structures and data in Azure Databricks, including notebooks, clusters, and catalogs
- Work with Unity Catalog, including schemas, tables, external locations, and data governance
- Optimize ingestion and transfer processes for large volumes of data using ADLS Gen2, AzCopy, Delta, and Parquet
- Manage notebook parameterization, including widgets, libraries, and cluster sizing
- Automate routines using Python, PySpark, and SQL
- Support validation, user acceptance testing, and troubleshooting in collaboration with technical teams and the vendor
- Ensure data integrity, performance, and governance in mission-critical environments
Requirements:
- Strong experience with Azure Databricks and PySpark/Python
- Knowledge of Azure Data Lake Storage Gen2, ACLs, and AzCopy
- Advanced proficiency in SQL and data engineering
- Experience with Unity Catalog and data governance
- Analytical, self-motivated professional with strong technical communication skills
Similar Jobs
Vice President, Data Engineering
Vice President leading data strategy, architecture, and engineering for PICTON Investments, a Canadian alternative investment manager. Building financial data platforms supporting portfolios, analytics, risk, and AI.
Director of Data Engineering
Director of Data Engineering leading scalable, governed data platforms for OnTrac’s same-day and next-day U.S. delivery services. Overseeing pipelines, AI-ready data products, budgets, and engineering teams.
Engagement Lead – Enterprise Data Warehouse
Engagement Lead owning CIO-level client relationships and prioritizing enterprise data warehouse delivery. Shaping Snowflake remediation and Power BI dashboards for Bounteous, a global AI services firm.
DBA, Bulk Data Management Engineer
DBA engineering Webbing’s global telecoms connectivity and IoT data platform. Managing PostgreSQL, telecom network databases, bulk data pipelines, governance, and disaster recovery.
DBA, Bulk Data Management Engineer
DBA engineering Webbing's global telecom connectivity and IoT platform across PostgreSQL, BSS, and network databases. Building resilient data operations, integrations, governance, and disaster recovery.
DBA, Bulk Data Management Engineer
DBA managing PostgreSQL, bulk data, and governance for Webbing’s global telecom connectivity and IoT platform. Building resilient databases, mediation pipelines, monitoring, and disaster recovery.
DBA, Bulk Data Management Engineer
DBA engineering PostgreSQL databases and bulk-data pipelines for Webbing’s global connectivity and IoT platform. Managing telecom network integrations, governance, high availability, and disaster recovery.
Data Engineer
Data Engineer diseñando pipelines, arquitecturas y plataformas de datos escalables para iCodde. Integrando, transformando y asegurando datos para análisis y decisiones operativas.
Collibra Engineer – Data Governance
Collibra Engineer improving data catalogs, metadata, and workflows for AMDEX.ai’s federal data modernization services. Supporting a DHS client with governance, quality, and interoperability.
Data Engineer, Active Secret Clearance
Data Engineer designing secure AWS GovCloud architectures and pipelines. Supporting GDIT’s mission-critical U.S. government, defense, and intelligence technology services.
Senior Data Engineer
Engenheiro de Dados Sr. construindo pipelines, Data Lakes e arquiteturas Lakehouse em Azure e Databricks para uma consultoria de transformação digital. Otimizando dados, IA e governança para grandes clientes.
Arquiteto de Dados e Soluções
Arquiteto de dados e soluções projetando arquiteturas Cloud, Lakehouse e DataOps. Digiage conecta APIs, dados e IA para transformar grandes empresas.
Data Engineer
Database Administrator scaling albo’s hybrid financial data infrastructure. Managing GCP, ETL pipelines, MySQL, MongoDB, and SPEI payment-system reliability.
Data Engineer – Data Platform
Data Engineer building scalable AWS data pipelines and platforms for INTERSPORT’s e-commerce operations. Enabling company-wide analytics, reporting, and data-driven decisions.
Data Engineer
Data Engineer building resilient data pipelines for Zeal, an AI-enabled software consultancy serving Fortune 1000 companies. Delivering migrations, transformations, governance, and production-ready data architectures.
Data Engineer II
Engenheiro de Dados desenvolvendo plataformas, pipelines ETL/ELT e soluções de qualidade de dados. Apoio a insights de clientes para a indústria automotiva e redes de concessionários da MSX.
Senior Data Engineer, Databricks
Senior Databricks Data Engineer building scalable data pipelines with Python and Databricks. Delivering cloud solutions across AWS, Azure, or GCP for Optiveum’s clients.
Senior Data Engineer
Senior Data Engineer designing secure data architectures and optimizing ETL/ELT pipelines. Supporting cloud migrations, governance, and batch and real-time analytics processing.
Data Engineer
Engenheiro de Dados desenvolvendo e sustentando pipelines corporativos em Azure e Databricks. Apoio à governança, qualidade e performance de dados na Cadmus.
Data Engineer
Data Engineer II building healthcare data pipelines and analytics for a Massachusetts-based consulting firm. Supporting healthcare quality, affordability, and equity through claims data solutions.