Senior Consultant – AI and Data, AWS Databricks
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
AWS Data Engineer developing scalable ETL pipelines in EY's D&A team for complex data solutions. Collaborating across various industries to modernize data ecosystems and analytics platforms.
Responsibilities:
- Develop, optimize, and deploy scalable ETL/ELT pipelines using PySpark, SQL, and AWS services.
- Good understanding in medallion architecture using Databricks lakehouse.
- Strong knowledge in Databricks end-to-end platform management (Unity Catalog, Users/Groups, security etc.), Databricks integrations (AWS/Azure/GCP.), DevOps – CI/CD pipelines (for notebooks, mlflows etc).
- Build and manage data lakehouse solutions leveraging AWS S3, Glue, Iceberg, and other AWS-native components.
- Migrate on-premises ETL workloads to modern AWS-based architectures with a focus on performance, reliability, and cost efficiency.
- Implement metadata-driven ingestion frameworks and medallion/layered architecture (Bronze/Silver/Gold).
- Work on orchestration frameworks such as Astronomer (Airflow), AWS Step Functions, or AWS-managed workflows.
- Follow data warehouse (DW) concepts and best practices for modeling and data integration.
- Collaborate with data analysts, data scientists, and BI engineers for seamless data delivery.
- Perform code reviews, troubleshoot issues, and ensure end-to-end data quality.
Requirements:
- 8+ years of total IT experience with at least 2+ years in AWS-based data engineering.
- Strong hands-on experience in Databricks Lakehouse Architecture.
- PySpark, Scala, SQL, Python ETL pipeline development.
- AWS services: S3, Glue, Lambda, Step Functions, CloudWatch.
- Unix/Linux environments and Iceberg table format.
- Astronomer (Airflow) or other orchestration tools.
- Experience with structured and semi-structured data formats such as Parquet, JSON, CSV, XML.
- Understanding of data warehousing concepts, star/snowflake schemas, and dimensional modeling.
- Good knowledge of CI/CD, version control (GitHub, Azure DevOps, Jenkins).
- Strong analytical, troubleshooting, and problem-solving skills.
- Ability to work independently and collaborate with stakeholders to deliver high quality solutions.
Benefits:
- Opportunities to work on diverse, meaningful, and industry-leading projects.
- Coaching, learning programs, and a personalized growth and development plan.
- Exposure to a collaborative, interdisciplinary work culture.
- Flexibility to manage your work in the way that suits you best.
- Supportive colleagues and a global environment for continuous knowledge exchange.
Similar Jobs
AI Tutor, Physics Specialist
Physics Specialist contributing expertise to AI model evaluation at Handshake AI. Developing and assessing challenging physics problems and critiquing AI-generated responses.
AI Tutor, Organic & Polymer Chemistry Specialist, NMR/Spectroscopy
AI Tutor specializing in Organic & Polymer Chemistry to assess AI model outputs for Handshake AI. Involves designing chemistry prompts and evaluating scientific reasoning of AI-generated responses.
AI Tutor, Electrochemistry, Functional Materials Specialist
Electrochemistry & Functional Materials Specialist contributing to AI model evaluation projects. Evaluating complex chemistry prompts and assessing AI-generated responses for scientific validity.
AI Tutor, Biophysical & Computational Chemistry Specialist
AI Tutor specializing in Biophysical & Computational Chemistry for AI model evaluations. Designing rigorous prompts and assessing model outputs in collaboration with a team of experts.
AI Tutor, Biology Specialist
AI Tutor specializing in biology, evaluating AI models through high-difficulty prompts. Collaborating with project leads to ensure quality and consistency in scientific evaluation.
VP of Operations, AI Simulation
VP of Operations responsible for operational strategies within the AI Simulation division at SandboxAQ. Leading high-impact teams to advance AI-driven science initiatives.
AI Copy Lead
Product Marketing Copywriter crafting exceptional copy for product launches and campaigns at ClickUp. Collaborating with teams to translate tech into engaging content.
Senior AI Governance Engineer
Sr. AI Governance Engineer at Cyclotron managing governance frameworks for enterprise clients using Microsoft AI. Designing effective strategies for Power Platform and Copilot Studio governance.
Associate Director, AI Transformation
Lead AI transformation initiatives at CHAI to improve operational workflows and staff productivity. Collaborate across teams to design, build, and deploy AI-powered tools for global health.
AI Chatbot Evaluator – Small Business Owners
AI Chatbot Evaluator participating in evaluation study using chatbot platforms. Small business owners engaging in prompt creation and response evaluation based on structured guidelines.
AI & Automation Freelancer
AI & Automation Freelancer for developing and implementing AI workflows and automation. Collaborating with internal teams to enhance AI infrastructure in German digital marketing company.
AI & Automation Manager – all genders
AI & Automation Manager at MAI Marketing Automation Intelligence. Supporting the operational execution of AI transformation and building AI workflows and systems.
Director of AI Strategy, Analytics
Director of AI Strategy and Analytics at Aimpoint Digital focusing on analytics and AI transformation. Leading strategy engagements, coaching teams, and overseeing complex projects.
AI & Automation Freelancer
AI & Automation Freelancer supporting the Head of AI at MAI Marketing. Building and optimizing AI infrastructure and automation workflows in a remote role.
AI & Automation Manager
AI & Automation Manager supporting the Head of AI in implementing AI workflows. Collaborating with teams to build automation systems and drive the AI transformation at MAI Marketing Automation Intelligence.
Senior Advisor – Artificial Intelligence
Senior Advisor leading executive-level projects in Artificial Intelligence at Cleveland Clinic. Partnering with stakeholders for organizational outcomes in a remote role across the U.S.
Forward Deployed Engineer, Data & AI
Forward Deployed Engineer building and deploying AI-driven solutions at Warner Music Group. Collaborating with stakeholders to streamline workflows and improve efficiency.
Staff Consultant – Gen AI
GenAI Engineer position requiring 3+ years in Python and implementation of GenAI solutions. Focus on collaboration, delivery, and optimization in a consulting environment.
Senior Consultant – Gen AI
GenAI Engineer focusing on developing RAG pipelines and secure APIs for enterprise-grade solutions at EY. Collaborating with diverse teams to deliver scalable AI services.
IT, AI Development Intern
IT/AI Development Intern providing hands-on exposure to software engineering and AI technologies at MIRA Safety. Collaborate on projects related to AI-powered tools, e-commerce, and analytics dashboard development.