Senior Data Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior Data Engineer building Python, Spark, and Databricks pipelines for Quantiphi’s data and AI solutions. Leading cloud migration, CI/CD automation, and scalable analytics engineering.
Responsibilities:
- Design, build, and optimize scalable real-time and batch data pipelines
- Use Python, SQL, PySpark, Spark Structured Streaming, Delta Live Tables, and Medallion Architecture
- Manage Databricks workflows, clusters, and Delta Lake
- Implement Unity Catalog, fine-grained access control, OAuth-based authentication, and enterprise security compliance
- Lead cloud migrations from Oracle, Teradata, and SQL Server to GCS, AWS S3, Azure Data Lake Storage, Snowflake, or Redshift
- Build CI/CD pipelines using Terraform and GitHub Actions
- Automate data pipeline deployments, infrastructure provisioning, and proactive monitoring
- Develop and deploy custom data-processing scripts and data-driven APIs using Python and Flask
- Use Docker for containerized processing
- Evaluate emerging technologies for data-intensive machine-learning tasks
- Collaborate with business stakeholders and cross-functional engineering teams to translate requirements into technical architectures and data-driven insights
- Guide projects and mentor team members
Requirements:
- 4–7 years of experience
- Notice period of 0–15 days; immediate joiners or candidates currently serving notice period preferred
- Hands-on experience with terabyte-scale datasets
- Experience designing distributed data processing solutions
- Advanced hands-on experience in Python, SQL, and PySpark
- Expert-level SQL query optimization for massive datasets
- Strong proficiency in Databricks Workflows, Unity Catalog, Delta Lake, and Delta Live Tables
- Deep familiarity with GCP, AWS S3/Redshift, or Azure ADLS Gen2
- Solid experience with Terraform, GitHub Actions, Docker, and CI/CD best practices
- Practical experience with Python web frameworks such as Flask
- Strong problem-solving, communication, and ownership skills
- Ability to guide projects and mentor team members
- Willingness to work 6:00 PM–3:00 AM IST during US business hours shift
Benefits:
- Culture built on transparency, diversity, integrity, learning and growth
- Environment encouraging innovation and professional and personal growth
Similar Jobs
IT Architect, Data Scientist
IT Architect/Data Scientist supporting DLH’s federal health and national-security programs. Leading cloud architecture, data science, interoperability, security engineering, and compliance for SCHARE.
Senior Business Data Engineer
Senior Business Data Engineer building AWS data infrastructure and automated reporting for Nex’s family motion-gaming platform. Improving pipeline reliability and supporting cross-functional business teams.
Senior Data Engineer – IBM DB2
Senior Data Engineer modernizing IBM DB2 data environments. Building reliable ETL/ELT pipelines and AWS integrations for enterprise applications and payroll systems.
Lead Data Architect
Lead Data Architect owning enterprise data architecture for C5MI Insight’s federal SAP S/4HANA modernization program. Leading data governance, migration, analytics architecture, and master data teams.
FBS Data Engineer, SQL, Power BI
Data Engineer supporting Farmers’ P&C insurance analytics and financial loss reporting solutions. Translating business data needs into requirements, integrations, dashboards, and validated reporting.
Data Engineer – Insurance
Data Engineer building analytics solutions, data pipelines, and enterprise integrations for Farmers Insurance. Improving data quality, reporting, dashboards, and regulatory-compliant business outcomes.
Power BI Data Engineer
Power BI Data Engineer building Azure data pipelines, dimensional models, and executive dashboards for Legence. Transforming complex data into actionable business intelligence across the full BI lifecycle.
Azure Data Engineer
Azure Data Engineer building Azure and Microsoft Fabric data solutions for Quisitive, a global Microsoft cloud partner. Delivering pipelines, integrations, and analytics projects for clients.
Senior Data Engineer
Senior Data Engineer building multi-region data infrastructure for Horizon3’s NodeZero autonomous cybersecurity platform. Supporting Federal workloads, warehouses, pipelines, orchestration, and BI reporting.
Data Engineer, Snowflake
Data Engineer Snowflake diseñando pipelines cloud y transformaciones ELT/ETL para Logicalis Spain. Participación en proyectos de ingeniería de datos, IA y analítica avanzada.
Data Engineer
Data Engineer building Python and AWS data pipelines for 540’s U.S. Army financial-system modernization initiative. Improving secure, scalable data transformation and governance.
Senior Manager, Data Engineering
Data engineering leader building Databricks and Azure platforms for Cushman & Wakefield, a global commercial real estate services firm. Managing engineers, platform delivery, governance, and stakeholder outcomes.
Data Architect
Data Architect designing Azure-integrated data platforms for Tieto’s agricultural analytics projects. Building scalable datasets, ETL pipelines, warehouses, and lakes.
Data Engineer
Data Engineer building scalable cloud pipelines, models, and enterprise data products for a major U.S. insurer. Supporting analytics, reporting, governance, and operational data capabilities.
Senior Data Engineer
Engenheiro de Dados SR desenvolvendo pipelines, Data Lakes e governança para o Grupo SysMap. Empresa brasileira de tecnologia acelerando a transformação digital de organizações.
Data Engineer
Data Platform Engineer owning cloud infrastructure, Terraform, CI/CD, and observability for Seer Interactive’s digital marketing consultancy. Automating analytics and AI platform operations.
Data Engineering Lead
Data Engineering Lead architecting Seer Interactive’s GCP data platform and agentic engineering workflows. Enabling governed analytics and AI products for digital marketing clients.
Data Engineer
Data Engineer building pipelines, warehouses, and analytics data for Joyce University’s healthcare education mission. Supporting reporting, compliance, accreditation, and institutional decision-making.
Enterprise Data Architect
Enterprise Data Architect designing Snowflake, ODS, and Kafka data architecture for Vista, a global business aviation company. Defining data models, pipelines, schemas, and governance standards.
Principal Data Engineer
Principal Data Engineer owning scalable pipelines and data-platform architecture for Everfi’s real-world digital learning solutions. Mentoring engineers and guiding cross-functional technical decisions.