Data Engineering Lead
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Data Engineering Lead overseeing design and development of next-generation data platform at MediaRadar. Spearheading data architecture and leading a team of data engineers for large-scale projects.
Responsibilities:
- Design and supervise the implementation of comprehensive data pipelines utilizing Azure Databricks and PySpark.
- Direct a team of data engineers, performing code reviews, offering technical expertise, and cultivating a culture of ongoing learning.
- Develop high-performance schemas in PostgreSQL and refine complex SQL queries for large datasets.
- Establish and apply optimal practices for data ingestion, transformation, and storage (Delta Lake/Lakehouse patterns).
- Collaborate closely with Data Analysts, Architects, and Product Managers to convert business requirements into technical specifications.
- Promote the implementation of CI/CD, unit testing, and automated monitoring to achieve 99.9% data reliability.
- Ensure data quality, governance, and compliance through validation, documentation, and secure practices.
- Continuously improve data systems for enhanced performance, reliability, and scalability.
- Effectively engage within an agile, cross-functional project team.
Requirements:
- Azure Databricks:
- ○ Expert-level experience managing workspaces, clusters, and job scheduling.
- ○ Solid understanding of data lakehouse architectures and Delta Lake.
- ○ Proven experience in Performance Tuning, Spark Optimization and Cost Reduction.
- PySpark: Advanced proficiency in Spark DataFrame APIs and Spark SQL for large-scale data processing involving various data formats.
- SQL Mastery: Exceptional ability to write, tune, and troubleshoot complex queries.
- PostgreSQL: Hands-on experience with relational database design, indexing, and performance optimization.
- ETL/ELT Frameworks: Proven track record of building scalable data pipelines from scratch.
- Workflow Orchestration: Experience with Apache Airflow for managing complex task dependencies.
- Containerization: Familiarity with Azure Kubernetes Service (AKS) for deploying containerized data services.
- Infrastructure as Code (IaC): Knowledge of Terraform or Bicep for managing Azure resources.
- 10+ years of experience in Data Engineering or Software Engineering.
- 3+ years as a formal technical Lead managing an agile team and implementing E2E solutions.
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
- Strong communication skills with the ability to explain complex technical concepts to non-technical stakeholders.
- Strong problem-solving skills and attention to detail.
Similar Jobs
Senior Data Engineering Analyst
Data Engineer managing data governance and modernization projects using Databricks at Keeggo. Collaborating with teams to enhance data pipeline architecture and performance.
Senior Data Engineer – AWS
Senior Data Analyst migrating data from SQL Server to AWS SaaS architecture, ensuring data integrity and developing reusable ETL processes.
Senior Data Engineer
Senior Data Engineer at Hyatt developing robust data pipelines and supporting data-driven decisions. Collaborating with cross-functional teams in a dynamic and fast-paced environment.
Senior Data Engineer
Senior Data Engineer designing, building, and maintaining data pipelines for global social discovery products. Collaborating with teams to turn data into insights and optimize data quality.
Staff Data Engineer
Staff Data Engineer at firsthand Health designing and optimizing data infrastructure. Engaging in impactful work supporting individuals living with serious mental illness with a modern technology stack.
Data Engineer, Databricks
Data Engineer overseeing data operations and solutions within Azure cloud environments. Collaborating on Databricks and ensuring reliable data processes and governance practices.
Data Architect – Tieto Tech Consulting
Data Architect designing scalable data architectures at Tieto Tech Consulting. Guiding clients through complex data solutions and fostering innovation in data and tech.
Lead Data Engineer – Tieto Tech Consulting
Lead Data Engineer at Tieto Tech Consulting, focusing on designing and evolving data platforms. Collaborating on data engineering initiatives and guiding project teams to deliver impactful solutions.
Data Architect
Data Architect designing scalable, secure architectures at Tieto Tech Consulting. Guiding complex decisions and collaborating with stakeholders for impactful data solutions.
Lead Data Engineer – Tieto Tech Consulting
Lead Data Engineer at Tieto Tech Consulting transforming raw data into impactful business solutions. Analyzing systems, guiding teams, and ensuring high-quality data architecture delivery.
Senior Data Architect – Common Data Model, Metadata Platforms
Senior Data Architect at a tech services company developing common data models and metadata platforms. Leading architecture design and integrating data systems for enterprise solutions.
Engenheiro de Dados – Sênior
Senior role in Data Engineering focusing on GCP with responsibilities in data governance, modeling, and pipeline management.
Data Engineer
Engenheiro de Dados criando dashboards e integrando dados de várias fontes. Colaborando com equipes de negócios e desenvolvendo soluções de BI de forma remota.
Senior Data Engineer
Data Engineer responsible for designing and maintaining scalable data pipelines at a global IT firm. Collaborating with various stakeholders to deliver insights from data solutions.
Data Engineer III
Data Engineer focusing on development and deployment of data-driven systems solutions at Raytheon. Analyzing ERP data, curating datasets, and collaborating with stakeholders to enhance business performance.
Data Engineer – 4-month contract
Data Engineer creating scalable data pipelines for autonomous truck technology. Focusing on ETL/ELT workflows and AWS-based data infrastructure management.
Senior Data Architect
Senior Data Architect supporting data architecture initiatives across cloud environments at Goods & Services. Designing, implementing, and optimizing data infrastructure and solutions to drive innovation and decision-making.
Data Architect
Data Architect designing scalable data solutions at EX Squared LATAM. Bridging business and technology to drive data-driven decision-making.
Data Engineer
Data Engineer at Anika Systems responsible for designing and building data pipelines. Supporting federal clients with data strategies and ensuring high-quality data for analytics.
Data Architect
Data Architect leading enterprise data architecture design and implementation for federal clients. Shaping data strategy and supporting data-driven decision-making with AI-driven workflows.