Data Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior Data Engineer at Pythian collaborating globally to design impactful cloud data solutions. Leading technical projects and optimizing data architectures with a focus on advanced analytics.
Responsibilities:
- Design and develop end-to-end cloud-based solutions with a strong emphasis on data applications and infrastructure.
- Lead discovery and design sessions with customers to gather requirements and translate functional needs into detailed designs.
- Create and contribute to technical design documents and other project-related documentation.
- Work with stakeholders to identify technical and business requirements, and apply best practices and standards to achieve successful project outcomes.
- Regularly demonstrate proficiency in established practices and standards for cloud solutions.
- Write high-performance, reliable, and maintainable code.
- Develop test automation frameworks and associated tooling to ensure project success.
- Handle complex and diverse cloud-based projects, including tasks such as collecting, managing, analyzing, and visualizing very large datasets.
- Build efficient and scalable data pipelines for batch and real-time use cases across various source and target systems.
- Optimize ETL/ELT pipelines, troubleshoot pipeline issues, and enhance observability dashboards.
- Execute data pipeline-specific DevOps activities, such as IaC provisioning, implementing data security, and automation.
- Analyze potential issues, perform root cause analyses, and resolve technical challenges.
- Review bug descriptions, functional requirements, and design documents to ensure comprehensive testing plans and cases.
- Performance tuning of batch and real-time data processing pipelines.
- Ensure security best practices are followed when working on internal and customer-facing cloud data platforms.
- Build foundational CI/CD pipelines for all infrastructure components, data pipelines, and custom data applications.
- Develop observability and data quality solutions for data platforms, including ML and AI applications.
- Act as a trusted advisor for customers, addressing technical queries and providing support.
- Engage in thought leadership activities such as whitepaper authoring, conference presentations, and podcasting.
- Suggest and implement ways to improve project progress and efficiency.
- Participate in pre-sales activities when required.
Requirements:
- Experience in implementing complex data architecture, data modeling, data design, and persistence (e.g., warehousing, data marts, data lakes).
- Proficiency in a programming language such as Python, Java, Go, or Scala.
- Experience with big data cloud technologies like Microsoft Fabric, Databricks, EMR, Athena, Glue, BigQuery, Dataproc, and Dataflow.
- Ideally, you will have specific strong hands-on experience working with Google Cloud Platform data technologies—Google BigQuery, Google DataFlow, and executing PySpark and SparkSQL code at Dataproc.
- Solid understanding of Spark (PySpark or SparkSQL), including using the DataFrame Application Programming Interface as well as analyzing and performance tuning Spark queries.
- Strong experience in data orchestration using Apache Airflow.
- Highly proficient in SQL.
- Strong experience in using code repositories such as GitHub and demonstrable GitOps best practices.
- Bring a good knowledge of popular database and data warehouse technologies and concepts from Google, Amazon, or Microsoft (Cloud & Conventional RDBMS), such as BigQuery, Redshift, Microsoft Azure SQL Data Warehouse, Snowflake, etc.
- Have knowledge of how to design distributed systems and the trade-offs involved.
- Have strong knowledge of CI/CD tools and frameworks such as Jenkins and GitLab to implement DevOps pipelines.
- Proficiency in using GenAI tools for productivity e.g. Copilot.
Benefits:
- Competitive total rewards package
- Blog during work hours; take a day off and volunteer for your favorite charity.
- Flexibly work remotely from your home, there’s no daily travel requirement to an office!
- All you need is a stable internet connection.
- Collaborate with some of the best and brightest in the industry!
- Hone your skills or learn new ones with our substantial training allowance; participate in professional development days, attend training, become certified, whatever you like!
- We give you all the equipment you need to work from home including a laptop with your choice of OS, and an annual budget to personalize your work environment!
- You will have an annual wellness budget to make yourself a priority (use it on gym memberships, massages, fitness and more).
- A generous amount of paid vacation and sick days, as well as a day off to volunteer for your favorite charity.
Similar Jobs
Data Architect
Data Architect evaluating and evolving data architecture for partners at EVT. Focus on ingestion, processing, and structuring of data with a collaborative team environment.
Data Engineer
Data Engineer responsible for designing, building, and maintaining data pipelines at Loopback Health. Collaborating with cross-functional teams to drive system and process improvements.
Data Engineer
Data Engineer optimizing and maintaining data pipelines in Blackline Safety's IoT-enabled safety ecosystem. Collaborating with product, engineering, and analytics teams on impactful data-driven initiatives.
Senior Ingeniero de Datos
Senior Data Engineer developing data solutions for transforming lives at Sofka. Collaborating in a close-knit team focused on continuous improvement and technical excellence.
Senior Data Engineer
Senior Data Engineer at Arsipa developing scalable data platforms and ETL/ELT pipelines. Collaboration with teams to ensure data quality, performance, and innovation in data architecture.
Data Engineer
Data Engineer shaping cloud-based data solutions for top U.S. companies. Engage with diverse projects remotely from Latin America.
Senior Data Engineer
Senior Technical Consultant delivering end-to-end data solutions and managing client relationships. Collaborating with technical team to drive project success with Microsoft technologies.
Senior Data Engineer – Data for Products
Lead the design and implementation of scalable batch and streaming data pipelines on Azure. Collaborate with product teams to ensure data quality, performance, and governance across the data lifecycle.
Data Engineer
Data Engineer maintaining scalable data pipelines for an international company in Hamburg. Requires experience with Snowflake, ETL processes and Azure Data Factory.
IS Data Warehouse Architect – III
Data Warehouse Architect designing and developing data models and ETL specifications. Collaborating on data architecture and vendor relations for a nonprofit health care organization.
Senior Director – Data Platform, Analytics
Leading the development of an integrated data platform at Harbor Compliance. Unifying fragmented data and establishing governance to empower decision-making.
Senior Data Engineer
Senior Data Engineer at D.A. Davidson optimizing data integrations and migrating legacy structures. Involves cloud architecture and coordination with product teams.
Data Architect
Professional Services Delivery Architect at Ciena guiding global delivery teams and ensuring high-quality outcomes for customer engagements. Leading complex transformations and influencing solution strategy.
Distinguished Architect, Data Platform
Distinguished Architect leading the design of a next-generation data platform at CloudZero. Driving technical decisions and architectural innovations for scalable cloud cost management solutions.
Senior Data Engineer
Data Engineer designing and maintaining data pipelines for Arrow Electronics. Collaborating with teams in engineering and architecture discussions across global projects.
Data Engineer – AI, Spark, Databricks, Healthcare
Data Engineer managing advanced Spark scripts for data validation and integration at Cotiviti. Collaborating with engineering teams to ensure data integrity and compliance in healthcare data processes.
Senior Software Engineer, Data Platform Team
Senior Software Engineer designing and developing features for data platforms at Sysdig. Leading technical strategy and mentoring teams in cloud security innovations.
Data Engineering Architect – Manager
Data Engineering Architect managing scalable AWS Data Lakehouse architectures at Naveera Technology. Overseeing data engineering projects with a focus on analytics and reporting solutions.
AI Data Engineer
AI Data Engineer bridging data warehousing and AI-driven analytics capabilities. Collaborating on automated solutions and operational workflows in eCommerce with global brands.
AI Data Engineer
AI Data Engineer at Netwrix designing and building enterprise-grade data and AI platforms. Focused on stability and repeatability while adhering to security and governance standards.