Lead Data Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Lead Data Engineer at Mercator.ai building scalable data infrastructure and distributed data pipelines. Ensuring performance and reliability while mentoring junior engineers.
Responsibilities:
- Lead the architecture and evolution of scalable, distributed data pipelines, ensuring high availability and performance at scale
- Design and implement robust data models to support reporting and advanced data applications
- Build and maintain distributed web scraping systems using tools such as Playwright, Selenium, and BeautifulSoup
- Develop systems capable of handling anti-scraping measures, proxy rotation, and high-volume data extraction
- Integrate AI and LLMs into engineering workflows for code generation, automation, and optimization
- Apply prompt engineering techniques to improve data processing, documentation, and troubleshooting
- Identify and implement system and process improvements to optimize performance and efficiency
- Manage and scale cloud-based data infrastructure, including data warehouses, object storage, and search systems
- Deploy and maintain containerized workloads using Kubernetes
- Implement data quality monitoring and governance processes to ensure accuracy and reliability
- Mentor junior engineers through code reviews, documentation, and knowledge sharing
- Communicate technical concepts clearly and provide business context for engineering decisions
Requirements:
- 5+ years of experience in Data Engineering with a track record of scaling systems
- Expert proficiency in Python and advanced SQL, including performance tuning and optimization
- Strong experience with workflow orchestration tools such as Airflow or Prefect and transformation tools such as dbt
- Proven experience building resilient web scraping systems using Playwright, Selenium, and BeautifulSoup
- Deep understanding of relational and NoSQL databases including Postgres, MongoDB, and ElasticSearch
- Experience working with large-scale data systems such as BigQuery
- Strong proficiency with CI/CD pipelines, Git, and Docker
- Experience designing and maintaining distributed systems with high availability and fault tolerance
Similar Jobs
Principal Data Architect
Principal Data Architect at Committee for Children, defining and evolving data architecture for a social enterprise. Collaborate with teams to ensure data governance and accessibility for child welfare programs.
Senior Data Engineer, Snowflake
Sr. Data Engineer designing and implementing scalable data solutions in Snowflake for clients. Collaborating with teams to translate requirements into actionable technical solutions and optimizations.
Data Engineer – DataOps, Infrastructure Focus
Data Engineer responsible for building and maintaining AWS data infrastructure for Guidewire. Collaborating with teams to support AI-driven analytics and operational excellence.
Data Architect III
Data Architect III managing data strategy and overseeing data collection and organization in a remote setting for Veterans Affairs. Leading data architecture and governance initiatives to improve Veteran services.
Senior Manager, Data Architecture
Senior Manager of Data Architecture ensuring enterprise data is standardized, secure, and well modeled. Overseeing data architecture across platforms and applications and translating business outcomes into data models.
Senior Data Engineer
Senior Data Engineer at Peek transforming data sources into actionable insights for analytics. Collaborating across teams to support business intelligence and enhance data infrastructure.
Senior Data Engineer
Senior Data Engineer building and scaling data infrastructure for Wagmo’s pet health solutions. Collaborate with multiple teams to ensure clean data flows across the business.
Senior Clinical Data Engineer
Senior Clinical Data Engineer overseeing clinical data lifecycle for Oura's research studies. Collaborating with teams to maintain data integrity and regulatory compliance in clinical trials.
Lead Data Engineer
Lead Data Engineer overseeing data platform and analytics engineering teams at OneOncology. Responsible for operational excellence and technical direction in data processing and management.
Senior Data Engineer
Senior Data Engineer at Kainos designing and developing data processing software for large-scale data solutions. Collaborating with agile teams to ensure high code quality and solution effectiveness.
Data Engineer
Data Engineer at Egen designing and building data processing pipelines using advanced technologies. Collaborating with stakeholders to ensure data accessibility, security, and accuracy.
Data Architect
Data Architect providing data management and analytics support for HRSA's Organ Procurement program. Collaborating on patient safety performance, technical assistance, and data visualization.
AI Data Engineering Lead
AI Data Engineering Lead building data foundations for AI models at Ideagen. Leading a team to ensure data quality, compliance, and operational efficiency.
Cloud Data Architect
Cloud Data Architect at Hunter Strategy responsible for telemetry and data normalization in cybersecurity. Designs log collection and routing architectures for AWS and Azure environments.
Senior Data Engineering Analyst
Data Engineer managing data governance and modernization projects using Databricks at Keeggo. Collaborating with teams to enhance data pipeline architecture and performance.
Senior Data Engineer – AWS
Senior Data Analyst migrating data from SQL Server to AWS SaaS architecture, ensuring data integrity and developing reusable ETL processes.
Senior Data Engineer
Senior Data Engineer at Hyatt developing robust data pipelines and supporting data-driven decisions. Collaborating with cross-functional teams in a dynamic and fast-paced environment.
Senior Data Engineer
Senior Data Engineer designing, building, and maintaining data pipelines for global social discovery products. Collaborating with teams to turn data into insights and optimize data quality.
Staff Data Engineer
Staff Data Engineer at firsthand Health designing and optimizing data infrastructure. Engaging in impactful work supporting individuals living with serious mental illness with a modern technology stack.
Data Engineer, Databricks
Data Engineer overseeing data operations and solutions within Azure cloud environments. Collaborating on Databricks and ensuring reliable data processes and governance practices.