Senior Data Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior Data Engineer developing scalable data solutions on the Databricks platform at CDW. Collaborating with architects and data science teams to deliver high-quality data products.
Responsibilities:
- Build and maintain scalable data pipelines, ETL and ELT processes, and data models within the Databricks platform.
- Design, develop, and deploy data and AI solutions using Databricks, Spark, Delta Lake, and related technologies.
- Develop batch and streaming pipelines using tools such as Databricks Workflows and Azure Data Factory.
- Design logical data flow diagrams and normalized schemas, implementing Lakehouse patterns such as the Medallion Architecture (Bronze, Silver, Gold layers).
- Ensure data quality, integrity, security, and governance throughout the data lifecycle, including use of Unity Catalog.
- Optimize Spark jobs and data transformations through effective partitioning, caching, and join strategies.
- Monitor pipeline execution, identify failures, and troubleshoot complex data processing issues.
- Collaborate with data architects, analysts, data scientists, and business stakeholders to understand requirements and deliver solutions.
- Support documentation of data processes, standards, and data flows.
Requirements:
- 5 Years of experience designing, developing, and deploying data solutions on the Databricks platform.
- Proficiency in Python, including PySpark, and SQL.
- Hands-on experience with Spark, Delta Lake, and Lakehouse architectures.
- Experience implementing data quality, governance, and security practices across data pipelines.
- Familiarity with machine learning concepts, tools, and libraries such as TensorFlow, PyTorch, Scikit-learn, and MLflow is a plus.
- Experience configuring and integrating external AI models and working with AI governance and monitoring tools is a plus.
- Experience with asynchronous programming patterns in Python for building scalable data or AI workloads is a plus.
- Strong problem-solving, collaboration, and communication skills.
Benefits:
- Health insurance
- 401(k) matching
- Paid time off
- Professional development
Similar Jobs
Senior Business Data Engineer
Senior Business Data Engineer designing, deploying, and maintaining data analytics pipeline for business insights. Collaborating with senior leadership and data teams to improve metrics and processes.
Engenheiro de Dados Pleno
Data Engineer responsible for ingesting data and developing cloud-based data pipelines. Collaborating with technical teams to optimize data processes and solutions.
Senior Data Engineer
Senior Data Engineer in India's remote role, focusing on AWS/GCP environments and ETL workflows optimization with responsibilities in data pipeline architecting.
VP, AI & Data Platform
Vice President leading AI and Data Platform architecture at ENFRA. Overseeing the delivery of a modern data and AI platform for competitive energy solutions.
Senior Data Engineer, ML Ops
Senior Data Engineer architecting the data platform at Path Robotics. Delivering scalable data pipelines to support machine learning and AI solutions in robotics.
Senior Scientific Data Engineer
Senior Research AI Scientist partnering with multi-disciplinary teams to operationalize AI capabilities for high-priority scientific needs. Working at Bristol Myers Squibb to pioneer transformative insights for patient benefit.
Principal Data Engineer
Principal Data Engineer at Northrop Grumman designing real-time reporting solutions as a SAP S/4HANA Analytics Developer. Collaborating on complex ABAP CDS views and integration with supply chain requirements.
Engenheiro de Dados Sênior
Engineering Data Engineer at Grupo SysMap developing scalable data pipelines in AWS, Azure, and GCP environments. Collaborating on modern data architectures and implementing data ingestion processes.
Engineering Program Manager – Data, Analytics, Data Migration
Engineering Program Manager leading cross-functional data and analytics projects in retail sector. Collaborating with teams on user behavior and analytics tools in a fast-paced environment.
Staff Data Engineer – Governance
Sr. Staff Data/Platform Engineer with expertise in distributed systems for AI-driven Data Platform. Contributes to scalability and real-time intelligence at Blip.
ML Data Engineer
ML Data Engineer responsible for designing and developing AI platforms at Newfold Digital. Collaborating across teams to integrate and optimize data sources for AI-driven applications.
Data Architect
Data Architect engaging with companies on transformational data programs to enhance AI and data capabilities. Leading architectural frameworks and mentoring data teams against industry best practices.
Senior Data Engineer, Data Science
Senior Data Analyst creating machine learning solutions and leading experiments to enhance client experience. Collaborating across teams to drive data-driven decisions and optimize results.
Data Engineer
Data Engineer architecting data flow for supply chain intelligence at Shipium. Designing ETL processes to optimize analytics, reporting, and drive AI products.
Data Engineer, Data Project Lead
Data Engineer leading data management and engineering projects supporting OHS program oversight. Ensuring compliance with federal policies while delivering high-quality data solutions.
Junior/Mid-level Data Engineer
Junior Data Engineer at Vitrio supporting data pipeline maintenance and data quality monitoring. Working with the Data Engineering team on Google Cloud Platform.
Senior Data Engineer
Senior Data Engineer focusing on data pipelines and collaboration with blockchain engineers to index blockchain data for the Solana ecosystem.
Senior Data Engineer
Data Engineer at Grupo SysMap focused on data modeling and pipeline development for multiple cloud environments. Analyzing business processes and ensuring data integrity with best practices.
Data Architect
Teamcenter Data Architect designing and configuring PLM data models at Accelint. Contributing to mission-ready PLM environments supporting Navy objectives with data consistency.
Lead Technology – Web Scraping, Data Platform
Technical leader in technology for a startup specializing in market intelligence and data monitoring. Leading technical decisions in web scraping, data infrastructure, and back-end development.