Senior Staff Data Engineer, Databricks
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior Staff Data Engineer building governed Databricks lakehouses for Shield AI, a defense-tech company developing autonomous aircraft and intelligent systems. Designing secure ingestion, quality, and deployment pipelines.
Responsibilities:
- Design and build batch and streaming ingestion pipelines from enterprise source systems into the Databricks lakehouse using Delta Lake
- Own Bronze-layer ingestion, including raw landing patterns, metadata capture, load traceability, and recoverable ingestion design
- Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement
- Define and evolve reusable ingestion and transformation patterns, templates, and engineering standards
- Implement and maintain Databricks platform constructs, including catalogs, schemas, service principals, job orchestration, and environment-aware deployment patterns
- Build and maintain CI/CD pipelines for ingestion code, transformation logic, workflow definitions, tests, and environment promotion across dev, test, and prod
- Apply data classification, segregation, and handling requirements within pipeline design
- Build data quality controls reflecting business meaning, record integrity, completeness, and expected domain behavior
- Quarantine, flag, and route problematic records or datasets according to quality and compliance rules
- Maintain documentation for source objects, ingestion logic, transformations, data quality rules, and known limitations
- Partner with the Analytics Engineer and domain teams to ensure reliable, governed Silver-layer data for Gold-layer modeling
- Collaborate with domain engineering teams on ownership boundaries, onboarding patterns, data contracts, and support expectations
Requirements:
- 12+ years of data engineering experience, including hands-on ownership of production data pipelines
- Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL
- Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts
- Experience with batch, CDC, and/or streaming ingestion patterns and associated operational trade-offs
- Experience with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled promotion across environments
- Strong SQL skills and solid grounding in data modeling fundamentals
- Demonstrated ability to understand the business and regulatory context of processed data
- Experience applying data classification, segregation, security, retention, or compliance requirements in regulated or security-sensitive environments
- Ability to design pipelines with awareness of data sensitivity, ownership, permitted use, and downstream impact
- Comfort operating in a fast-moving platform build and shaping engineering standards
- Databricks certification such as Data Engineer Associate or Professional (preferred)
- Experience standing up or maturing Unity Catalog structures and access models in an enterprise setting (preferred)
- Experience with CI/CD, infrastructure-as-code, and deployment automation for data platforms (preferred)
- Experience in defense, aerospace, financial services, healthcare, or another highly regulated environment (preferred)
- Experience enabling multiple business domains on a shared data platform while maintaining governance and ownership boundaries (preferred)
- Must be authorized to work in the United States
- Must satisfy applicable export-control requirements, such as being a United States citizen, lawful permanent resident, or protected individual
Benefits:
- Bonus
- Benefits
- Equity
- Temporary benefits package applicable after 60 days of employment for temporary employees
- Full-time regular employee offer package includes pay within range listed, bonus, benefits, and equity
Similar Jobs
Director, Technical Product Management – Data Platform
Director leading Alignment Health’s enterprise healthcare data platform for Medicare Advantage operations. Driving compliant data products, governance, roadmap delivery, and AI/ML enablement.
Senior Data Engineer
Senior Data Engineer designing scalable data pipelines and ETL for a product-led scaling bank. Contract role delivered by FCamara Europe across eligible EU countries.
Senior DataOps Engineer
Senior DataOps Engineer building regulated cloud data platforms and pipelines for Voyager Technologies’ space, defense, and national security missions. Leading AWS, Azure, Fabric, and DataOps modernization.
Senior Payroll Data Migration Specialist, Workday
Senior Workday payroll migration specialist coordinating multi-country data migrations for Spyrosoft, a software engineering company. Ensuring accurate, compliant payroll data through cleansing, ETL, reconciliation, and validation.
Payroll Data Migration Specialist
Payroll data migration specialist coordinating multi-country Workday migrations for Spyrosoft’s HR and payroll transformation projects. Ensuring accurate, compliant data from planning through validation.
Senior Engineer – Data Engineering, Databricks
Senior Databricks data engineer building scalable batch and streaming lakehouse pipelines. Delivering data platforms and solution accelerators for Slalom consulting clients.
Senior Data Engineer
Senior Data Engineer building scalable Databricks data pipelines and enterprise integrations. Supporting Sumo Logic’s platform that unifies security and operational data for reliable digital operations.
Senior Data Engineer
Senior Data Engineer building AWS and Python data pipelines for Dev.Pro, a US-based software development company. Integrating POS systems with a hotel chain’s data platform and reporting solutions.
Data Engineer, Celonis
Celonis Data Engineer building SAP-connected process-mining data models for Talan’s automotive clients. Designing pipelines, OCPM event logs, and Celonis architecture for Process Intelligence initiatives.
Senior Pharmacovigilance Data Migration Consultant – Oracle Argus Safety
Pharmacovigilance data migration consultant converting PostgreSQL safety systems to Oracle Argus for life sciences clients. Supporting validation, compliance, integrations, and database modernization.
Data Engineer
Data Engineer building Snowflake, dbt, Airflow, and AWS pipelines for IDEXX’s medical data products. Enabling trusted clinical insights for analysts and veterinary health teams.
AWS Data Architect
AWS Data Architect designing S3 Data Lakes, ETL frameworks, and analytics platforms for Leighton’s cloud consultancy. Enabling governed, secure, scalable AWS data solutions.
Senior Golang Engineer, Data Processing
Senior Golang Engineer building high-throughput Go pipelines for Litmus, whose platform turns industrial machine data into usable real-time AI infrastructure. Owning streaming services across cloud and edge environments.
Senior Data Engineer
Senior Data Engineer scaling data pipelines and MLOps infrastructure for Keebo’s cloud warehouse optimization platform. Building reliable models, features, and AI-enabled engineering workflows.
Senior Data Engineer, Ads Data – AI Native
Senior Data Engineer owning Databricks pipelines, metrics, and governance for Life360’s advertising data. Improving performance, cost, reliability, and self-serve insights for teams supporting family safety products.
Senior Data Engineer
Senior Data Engineer building reliable data pipelines and models for LMI’s federal national security applications. Supporting AI, analytics, deployment, and mission-user workflows.
Staff Data Engineer, CBP
Staff Data Engineer building scalable data pipelines, models, and ontology for CBP operations. Supporting LMI’s mission-ready digital solutions in Google Cloud.
Senior Contract Software Engineer, Data & Platform
Senior contract engineer building scalable data pipelines and platform services for Nimble’s software that helps schools recruit educators. Designing reliable systems and data-intensive products for education leaders.
Senior Data Engineer, Network Platform
Senior Data Engineer building scalable, governed data platforms for Cisco’s cloud-managed networking solutions. Developing reliable pipelines, data context layers, and AI-enabled engineering workflows.
Snowflake Data Architect
Senior Snowflake Data Architect designing Snowflake healthcare data platforms for Datavail. Leading patient identity, dimensional modeling, quality controls, and engineering delivery.