Senior Data Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Engenheiro de Dados Sr. construindo pipelines, Data Lakes e arquiteturas Lakehouse em Azure e Databricks para uma consultoria de transformação digital. Otimizando dados, IA e governança para grandes clientes.
Responsibilities:
- Design, develop and evolve scalable data pipelines using services from the Azure ecosystem and Databricks
- Develop robust data ingestion, transformation and provisioning processes (ETL/ELT), including real-time ingestion scenarios
- Implement and optimize solutions using Azure Data Factory, Azure Data Lake Storage (ADLS Gen2), Databricks Jobs/Workflows, Azure Functions, Azure Synapse Analytics and Event Hubs/Event Grid
- Design, implement and maintain Data Lakes and Lakehouse architectures (medallion: Bronze/Silver/Gold) on Azure Databricks
- Develop solutions using Python, SQL and PySpark with a focus on performance, scalability and data quality
- Work with large volumes of structured and unstructured data
- Ensure data quality, integrity, security and governance across the entire pipeline, leveraging Unity Catalog and compliance with LGPD
- Optimize queries, data processing and data consumption to reduce costs and improve performance
- Collaborate with Analytics, Data Science, Software Engineering and Architecture teams to build strategic solutions
- Implement monitoring, observability and failure handling in data pipelines
- Participate in defining standards, best practices and the evolution of the platform's data architecture
- Support analysis and resolution of critical incidents related to the data environment
Requirements:
- Proven experience as a Data Engineer in Azure environments
- Strong knowledge of core Azure data services such as Data Factory, ADLS Gen2, Databricks, Synapse Analytics and Azure Functions
- Solid experience developing ETL/ELT pipelines
- Advanced skills in Python and SQL
- Experience with data modeling for analytical environments and Data Lakes/Lakehouse
- Knowledge of Apache Spark or PySpark
- Experience with version control using Git and CI/CD practices (ideally Azure DevOps)
- Knowledge of partitioning, query optimization and distributed processing
- Experience with monitoring, observability and troubleshooting of pipelines
- Knowledge of cloud data architecture and best practices for security, governance and compliance with LGPD
- Strong analytical skills, problem-solving and ability to work in collaborative environments
- Nice to have: Experience with Databricks Unity Catalog for data governance and cataloging
- Nice to have: Knowledge of Delta Lake and medallion architecture (Bronze/Silver/Gold)
- Nice to have: Experience with Azure Data Factory and/or Databricks Workflows / Databricks Asset Bundles (DABs) for orchestration
- Nice to have: Experience with event-driven architectures (Event Hubs, Event Grid)
- Nice to have: Experience integrating legacy/ERP systems (e.g., SAP) into data pipelines
- Nice to have: Knowledge of data quality frameworks (e.g., DQX - Databricks Labs, dbt, Great Expectations)
- Nice to have: Experience with real-time data ingestion (e.g., Event Hubs, Kafka)
- Nice to have: Experience with Docker and Kubernetes (AKS)
- Nice to have: Knowledge of infrastructure-as-code (Terraform or Bicep/ARM Templates)
- Nice to have: Experience in Machine Learning, AI and MLOps projects
- Nice to have: Experience in large enterprise environments and mission-critical projects
- Nice to have: Azure and/or Databricks certifications (e.g., DP-203, Databricks Certified Data Engineer)
Benefits:
- Contract type can be PJ (contractor, without benefits) or CLT (employee)
Similar Jobs
Collibra Engineer – Data Governance
Collibra Engineer improving data catalogs, metadata, and workflows for AMDEX.ai’s federal data modernization services. Supporting a DHS client with governance, quality, and interoperability.
Data Engineer, Active Secret Clearance
Data Engineer designing secure AWS GovCloud architectures and pipelines. Supporting GDIT’s mission-critical U.S. government, defense, and intelligence technology services.
Arquiteto de Dados e Soluções
Arquiteto de dados e soluções projetando arquiteturas Cloud, Lakehouse e DataOps. Digiage conecta APIs, dados e IA para transformar grandes empresas.
Data Engineer
Database Administrator scaling albo’s hybrid financial data infrastructure. Managing GCP, ETL pipelines, MySQL, MongoDB, and SPEI payment-system reliability.
Data Engineer – Data Platform
Data Engineer building scalable AWS data pipelines and platforms for INTERSPORT’s e-commerce operations. Enabling company-wide analytics, reporting, and data-driven decisions.
Data Engineer
Data Engineer building resilient data pipelines for Zeal, an AI-enabled software consultancy serving Fortune 1000 companies. Delivering migrations, transformations, governance, and production-ready data architectures.
Data Engineer II
Engenheiro de Dados desenvolvendo plataformas, pipelines ETL/ELT e soluções de qualidade de dados. Apoio a insights de clientes para a indústria automotiva e redes de concessionários da MSX.
Senior Data Engineer, Databricks
Senior Databricks Data Engineer building scalable data pipelines with Python and Databricks. Delivering cloud solutions across AWS, Azure, or GCP for Optiveum’s clients.
Senior Data Engineer
Senior Data Engineer designing secure data architectures and optimizing ETL/ELT pipelines. Supporting cloud migrations, governance, and batch and real-time analytics processing.
Data Engineer
Engenheiro de Dados desenvolvendo e sustentando pipelines corporativos em Azure e Databricks. Apoio à governança, qualidade e performance de dados na Cadmus.
Data Engineer
Data Engineer II building healthcare data pipelines and analytics for a Massachusetts-based consulting firm. Supporting healthcare quality, affordability, and equity through claims data solutions.
Senior Data Engineer
Senior Data Engineer building Progress Partners’ foundational Lakehouse for global SaaS and mobile platforms. Designing AWS/GCP pipelines across payments, marketing, and operational data.
Data Engineer Manager
Data Engineering Manager leading data and ML infrastructure for Blend, an AI services provider. Building scalable cloud MLOps and foundations for future agentic AI initiatives.
Data Engineer Manager
Data Engineering Manager leading data and ML infrastructure for Blend, an AI services provider. Building scalable cloud MLOps and foundations for production machine learning and agentic AI initiatives.
Data Engineer
Data Engineer building trusted data pipelines, Airflow workflows, and dbt models for Grüns’ convenient whole-food nutrition gummies. Owning production data infrastructure and quality.
Senior Data Engineer
Senior Data Engineer building scalable, AI-powered data infrastructure for CloudBlue, HostPapa’s cloud commerce platform. Developing real-time pipelines, APIs, and production analytics systems.
Data Warehouse Developer
Data Warehouse Developer building Snowflake pipelines, integrations, and automation for Atrium Hospitality’s hotel operations. Supporting enterprise reporting and AI infrastructure with Python and SQL.
Product Owner – Cloud Contact Center, Data Platform
Product Owner leading cloud contact center and data platform delivery at BETSOL. Owning backlogs, user stories, PI planning, and enterprise prioritization for CCaaS transformation.
Staff Infrastructure Engineer – Data Streaming
Staff Infrastructure Engineer operating massive-scale Kafka and Redis data platforms for SentinelOne’s AI-native cybersecurity platform. Building portable, automated infrastructure across Kubernetes, multi-cloud, and air-gapped environments.
AWS Data Engineer
AWS Data Engineer joining Devoteam, a European digital strategy and technology consultancy. Building AWS data solutions with Spark, Python, SQL, Airflow, Docker, and ECS from Spain.