Data Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Data Engineer at Particle41 responsible for building data infrastructure and collaborating on data solutions. Contributing to projects while ensuring data accuracy, security, and performance.
Responsibilities:
- Design, develop, and maintain scalable ETL (Extract, Transform, Load) pipelines to process large volumes of data from diverse sources.
- Build and optimize data storage solutions, such as data lakes and data warehouses, to ensure efficient data retrieval and processing.
- Integrate structured and unstructured data from various internal and external systems to create a unified view for analysis.
- Ensure data accuracy, consistency, and completeness through rigorous validation, cleansing, and transformation processes.
- Maintain comprehensive documentation for data processes, tools, and systems while promoting best practices for efficient workflows.
- Collaborate with product managers, and other stakeholders to gather requirements and translate them into technical solutions.
- Participate in requirement analysis sessions to understand business needs and user requirements.
- Provide technical insights and recommendations during the requirements-gathering process.
- Participate in Agile development processes, including sprint planning, daily stand-ups, and sprint reviews.
- Work closely with Agile teams to deliver software solutions on time and within scope.
- Adapt to changing priorities and requirements in a fast-paced Agile environment.
- Conduct thorough testing and debugging to ensure the reliability, security, and performance of applications.
- Write unit tests and validate the functionality of developed features and individual elements.
- Writing integration tests to ensure different elements within a given application function as intended and meet desired requirements.
- Identify and resolve software defects, code smells, and performance bottlenecks.
- Stay updated with the latest technologies and trends in full-stack development.
- Propose innovative solutions to improve the performance, security, scalability, and maintainability of applications.
- Continuously seek opportunities to optimize and refactor existing codebase for better efficiency.
- Stay up-to-date with cloud platforms such as AWS, Azure, or Google Cloud Platform.
- Collaborate effectively with cross-functional teams, including testers, and product managers.
- Foster a collaborative and inclusive work environment where ideas are shared and valued.
Requirements:
- Bachelor's degree in Computer Science, Engineering, or related field.
- Proven experience as a Data Engineer, with a minimum of 3 years of experience.
- Proficiency in Python programming language.
- Experience with database technologies such as SQL (e.g., MySQL, PostgreSQL) and NoSQL (e.g., MongoDB) databases.
- Strong understanding of Programming Libraries/Frameworks and technologies such as Flask, API frameworks, datawarehousing/lakehouse principles, database and ORM, data analysis databricks, panda's, Spark, Pyspark, Machine learning, OpenCV, scikit-learn.
- Utilities & Tools: logging, requests, subprocess, regex, pytest
- ELK stack, Redis, distributed task queues
- Strong understanding of data warehousing/lakehousing principles and concurrent/parallel processing concepts.
- Familiarity with at least one cloud data engineering stack (Azure, AWS, or GCP) and the ability to quickly learn and adapt to new ETL/ELT tools across various cloud providers.
- Familiarity with version control systems like Git and collaborative development workflows.
- Competence in working on Linux OS and creating shell scripts.
- Solid understanding of software engineering principles, design patterns, and best practices.
- Excellent problem-solving and analytical skills, with a keen attention to detail.
- Effective communication skills, both written and verbal, and the ability to collaborate in a team environment.
- Adaptability and willingness to learn new technologies and tools as needed.
Benefits:
- Equal employment opportunities to all employees and applicants
- Assistance during the application or interview process
- Opportunity to work in a supportive and dynamic environment
Similar Jobs
Data Architect
Data Architect evaluating and evolving data architecture for partners at EVT. Focus on ingestion, processing, and structuring of data with a collaborative team environment.
Data Engineer
Data Engineer responsible for designing, building, and maintaining data pipelines at Loopback Health. Collaborating with cross-functional teams to drive system and process improvements.
Data Engineer
Data Engineer optimizing and maintaining data pipelines in Blackline Safety's IoT-enabled safety ecosystem. Collaborating with product, engineering, and analytics teams on impactful data-driven initiatives.
Senior Ingeniero de Datos
Senior Data Engineer developing data solutions for transforming lives at Sofka. Collaborating in a close-knit team focused on continuous improvement and technical excellence.
Senior Data Engineer
Senior Data Engineer at Arsipa developing scalable data platforms and ETL/ELT pipelines. Collaboration with teams to ensure data quality, performance, and innovation in data architecture.
Data Engineer
Data Engineer shaping cloud-based data solutions for top U.S. companies. Engage with diverse projects remotely from Latin America.
Senior Data Engineer
Senior Technical Consultant delivering end-to-end data solutions and managing client relationships. Collaborating with technical team to drive project success with Microsoft technologies.
Senior Data Engineer – Data for Products
Lead the design and implementation of scalable batch and streaming data pipelines on Azure. Collaborate with product teams to ensure data quality, performance, and governance across the data lifecycle.
Data Engineer
Data Engineer maintaining scalable data pipelines for an international company in Hamburg. Requires experience with Snowflake, ETL processes and Azure Data Factory.
IS Data Warehouse Architect – III
Data Warehouse Architect designing and developing data models and ETL specifications. Collaborating on data architecture and vendor relations for a nonprofit health care organization.
Senior Director – Data Platform, Analytics
Leading the development of an integrated data platform at Harbor Compliance. Unifying fragmented data and establishing governance to empower decision-making.
Senior Data Engineer
Senior Data Engineer at D.A. Davidson optimizing data integrations and migrating legacy structures. Involves cloud architecture and coordination with product teams.
Data Architect
Professional Services Delivery Architect at Ciena guiding global delivery teams and ensuring high-quality outcomes for customer engagements. Leading complex transformations and influencing solution strategy.
Distinguished Architect, Data Platform
Distinguished Architect leading the design of a next-generation data platform at CloudZero. Driving technical decisions and architectural innovations for scalable cloud cost management solutions.
Senior Data Engineer
Data Engineer designing and maintaining data pipelines for Arrow Electronics. Collaborating with teams in engineering and architecture discussions across global projects.
Data Engineer – AI, Spark, Databricks, Healthcare
Data Engineer managing advanced Spark scripts for data validation and integration at Cotiviti. Collaborating with engineering teams to ensure data integrity and compliance in healthcare data processes.
Senior Software Engineer, Data Platform Team
Senior Software Engineer designing and developing features for data platforms at Sysdig. Leading technical strategy and mentoring teams in cloud security innovations.
Data Engineering Architect – Manager
Data Engineering Architect managing scalable AWS Data Lakehouse architectures at Naveera Technology. Overseeing data engineering projects with a focus on analytics and reporting solutions.
AI Data Engineer
AI Data Engineer bridging data warehousing and AI-driven analytics capabilities. Collaborating on automated solutions and operational workflows in eCommerce with global brands.
AI Data Engineer
AI Data Engineer at Netwrix designing and building enterprise-grade data and AI platforms. Focused on stability and repeatability while adhering to security and governance standards.