Lead Data Engineer – PySpark, Palantir Foundry
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Lead Data Engineer building governed PySpark and Palantir Foundry pipelines for Logic20/20’s AI and analytics consulting practice. Improving releases, data infrastructure, and audit readiness for client projects.
Responsibilities:
- Deliver client value and ensure high client satisfaction
- Design, enhance, and maintain production-grade data pipelines supporting model output aggregation and downstream risk analysis
- Establish and mature repository governance practices, including branching strategies, pull request standards, merge policies, release tagging, and version control workflows
- Own release engineering practices for reproducible, traceable, and auditable production releases
- Refactor and improve pipeline code for modularity, maintainability, scalability, and documentation quality
- Develop configuration-driven pipeline patterns across environments and releases
- Support testing, validation, benchmarking, and change management for critical data pipelines
- Partner with data scientists, machine learning engineers, data engineers, product stakeholders, and other technical teams
- Align teams on schemas, interfaces, inputs, and delivery expectations
- Translate complex technical concepts into clear updates for technical and non-technical stakeholders
- Improve structure and governance in codebases, repositories, and engineering workflows
- Contribute to engineering best practices in a regulated, audit-sensitive delivery environment
Requirements:
- 10-15+ years of data engineering, data science, machine learning engineering, and/or relevant experience using Python
- Experience leading technical teams and overseeing enterprise-scale data initiatives
- Strong expertise in PySpark, SQL, and cloud services
- Ability to improve, refactor, or stabilize existing codebases and pipeline environments
- Experience with cloud-optimized datasets, efficient partitioning strategies, and large-scale spatial operations
- Understanding of machine learning model outputs flowing into downstream data pipelines, platforms, or production systems
- Experience in highly regulated industries such as utilities, financial services, healthcare, insurance, or similar environments
- Experience designing maintainable, scalable, and well-documented cloud-based data infrastructure or modern data platform environments
- Experience supporting reproducibility, dataset versioning, release traceability, and audit readiness
- Ability to define expected inputs, outputs, schemas, and interfaces across technical teams
- Strong communication skills
- Detail-oriented, governance-minded approach to engineering
- Practical experience with Git-based workflows, code reviews, branching strategies, and release management
- Experience with Palantir Foundry is highly preferred
- Experience with GIS technologies and geospatial data platforms
Benefits:
- Competitive base salary
- Performance-based bonuses
- Other incentives
- Training and mentorship opportunities
- Project opportunities for career development
- Supportive, globally connected work environment
Similar Jobs
Lead Geospatial Data Engineer
Lead geospatial data engineer guiding cloud-based geospatial, imagery, and enterprise data solutions at Logic20/20. Leading architecture, delivery, code quality, and engineering teams.
Senior Manager, Data Engineering – Medicare Revenue
Senior Manager owning CVS Health’s Medicare revenue data engineering foundation. Building auditable healthcare data pipelines for revenue forecasting, risk adjustment, Finance, and Actuarial teams.
Senior Data Engineer
Senior Staff Data Engineer leading Capital One’s enterprise data pipelines and cloud architecture. Driving scalable data platforms, governance, and innovation across banking technology.
Data Engineer III
Data Engineer III building AWS data lakes, pipelines, and analytics systems for Republic Services’ environmental services business. Developing MarkLogic APIs, Snowflake warehouses, and streaming ingestion architectures.
Senior Data Platform Engineer – Snowflake
Senior Data Platform Engineer bei JobRad Loop, Aufbau einer Snowflake-Datenplattform und neuer BI-Landschaft. Integration von JobRad-Leasingdaten, stabilen Pipelines und fachlicher Teamführung.
Data Engineer – Mid-Level
Mid-level Data Engineer building AI/ML data systems for Ignite Digital’s national security and Navy IT clients. Designing pipelines, models, dashboards, and integrations using Python, SQL, and QLIK Advana.
Senior Data Platform Engineer
Senior Data Platform Engineer building DEFCON AI’s durable, explainable data platform for resilient optimization. Preserving provenance, uncertainty, source disagreements, and point-in-time entity histories.
Senior Data Engineer
Senior Data Engineer designing banking data pipelines across AWS, Oracle, Airflow, dbt, and SAP. Modernizing legacy systems through Data Mesh and Medallion architectures.
Senior Data Engineer
Senior Data Engineer building BigQuery and Dataform pipelines for Avenga, an AI-focused international engineering firm. Migrating ETL workflows and modernizing analytics across cloud and on-premises systems.
Senior Data Engineer – Clinical Platforms, Databricks
Senior Data Engineer building Databricks-based clinical trial platforms at Muttdata. Designing compliant lakehouse pipelines and integrations for regulated clinical data.
Senior Big Data Architect
Senior Big Data Architect designing Cloudera, Kafka, and AWS data platforms for banking clients. Leading enterprise-scale streaming, integration, governance, and architecture standards.
Senior EDW Data Architect
Senior EDW Data Architect designing Teradata enterprise data warehouses for regulated banking clients. Leading governance, data modelling, SQL validation, and delivery support in a remote EU contract.
Data Architect – Microsoft Fabric
Microsoft Fabric Data Architect designing secure enterprise data platforms for Argano, a digital consultancy. Leading Fabric landing zones, analytics, governance, and generative AI architecture.
Senior Data Platform Engineer, Enterprise Merchant Integrations
Senior Data Platform Engineer building reliable AWS merchant integrations for Extend’s AI-driven post-purchase retail platform. Developing event-driven data systems, onboarding tooling, and error reporting.
Senior Data Engineer
Senior Data Engineer building ingestion, data quality, and graph infrastructure for Inflect’s digital infrastructure marketplace. Scaling InfraSpec and RouteFinder platforms across global teams.
Data Engineer – Reporting & Analytics
Data Engineer modernizing enterprise reporting and analytics for a global technology team. Building Tableau solutions, SQL datasets, pipelines, and cloud data platforms.
Software Engineer – Research Data Platform
Software Engineer building scalable data systems and APIs for Bioptimus’s universal biology AI foundation model. Optimizing storage, databases, and infrastructure for research and product teams.
Data Engineer, Snowflake
Senior Backend/Data Engineer building scalable Python, Django, and Snowflake data platforms. Supporting Addepto’s AI consulting and data engineering solutions for global enterprises and startups.
Master’s Scholarship Fellow, Data Engineering
Bolsista de mestrado pesquisando timelines inteligentes, eventos temporais e engenharia de dados para o Itaú. Desenvolvendo arquiteturas e protótipos de contexto para oportunidades baseadas em eventos.
Master’s Scholarship Researcher – Customer Timeline, Temporal Events, Data Engineering
Master’s researcher developing temporal data and event-driven architectures for Itaú’s intelligent customer timeline. Supporting prototypes for event-based opportunities and customer journeys.