Data Engineer III
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Data Engineer III designing and building complex data pipelines for AI/ML capabilities in the leading firm Robert Half. Collaborating with cross-functional teams to optimize data infrastructure and ensure data quality.
Responsibilities:
- Lead architecture and design of complex data pipelines on Databricks lakehouse architecture (Unity Catalog, Delta Lake, Structured Streaming)
- Define technical approach for data engineering initiatives, mentor less-senior engineers, and set standards for code quality through leadership and code reviews
- Design and build data foundations that enable AI/ML capabilities — feature stores, embedding pipelines, vector search indexes, and model training datasets
- Align data engineering solutions with business strategy, including support for Agentic AI workloads
- Own health, scalability, and modernization of data infrastructure with Databricks as the strategic platform — including workload migration, compute optimization, and Unity Catalog adoption
- Optimize pipeline performance (Delta Lake table layouts, clustering, Z-ordering) and establish monitoring/alerting best practices with clear SLAs
- Build data infrastructure supporting Agentic AI systems — real-time data access layers, context retrieval pipelines, and agent-accessible data services
- Collaborate cross-functionally with DevOps, Platform Engineering, and MLOps roles to integrate data solutions into the broader technology environment and shared AI infrastructure – Mlflow registries, feature stores, and agent orchestration layers
- Provide consultation to Senior Leadership on complex projects and drive continuous improvement initiatives
- Champion data governance at all layers for data, models, and AI assets
- Implement data quality strategies (master data management, validation rules, Delta Live Tables expectations) to ensure trust in enterprise data
- Serve as liaison across data engineering, AI engineering, and business teams; promote data literacy and stewardship
Requirements:
- Bachelor's in Computer Science, Engineering, or related field (Master's preferred)
- 5+ years with Python and SQL in data engineering for big data ML/analytics workloads
- 5+ years designing, building, and troubleshooting scalable ETL/ELT pipelines for business-critical production systems
- 3+ years with cloud data services (AWS), container orchestration (Docker, Kubernetes), and IaC (Terraform, CloudFormation)
- 3+ years architecting ML workflows and data platforms with CI/CD, automated testing, and distributed processing (Spark)
- 3+ years collaborating cross-functionally with Data Science, MLOps, Platform Engineering, and DevOps teams
- 3+ years implementing data quality testing and optimizing SQL/Python for cost/performance in the cloud
- Understanding of the full Data Science SDLC, and experience mentoring engineers
- Strongly Preferred - Databricks & AI Platform
- 2+ years hands-on with Databricks (Delta Lake, Unity Catalog, Databricks SQL)
- Experience with MLflow experiment tracking and model registry workflows
- Experience designing pipelines that serve AI/ML inference — real-time feature engineering, embedding generation, and context retrieval for LLM-based systems
- Understanding of how data engineering supports Agentic AI: agent-accessible data services, low-latency retrieval, and pipelines enabling autonomous multi-step workflows
- Familiarity with Databricks Mosaic AI, Vector Search, and/or Feature Store
- FinOps awareness — compute cluster optimization, cost attribution by workload
- Familiarity with Salesforce/Heroku data infrastructures
- Experience with data virtualization (e.g., Dremio)
- Understanding of Platform Engineering concepts and internal developer platforms
- Experience migrating from legacy data warehouse/lake to unified lakehouse architecture
- Familiarity with Odaseva data security and management
Benefits:
- group health insurance benefits (medical, vision, dental)
- FSA and HSA healthcare accounts
- life and accident insurance
- adoption and fertility assistance
- paid parental leave of up to 6 weeks
- short/long term disability
- paid time off for vacation, personal needs, and sick time
- up to 17 days of Choice Time Off (CTO) per calendar year
- up to 11 paid holidays per calendar year
- opportunity to contribute to company's 401(k) savings and investment plan or deferred compensation plan with an employer match of 100% on the first 3% of contributions
Similar Jobs
Data Engineer
Data Engineer responsible for designing and maintaining data architecture and infrastructure. Developing, testing, and deploying data solutions to meet organizational needs.
Data Engineer, L2
Data Engineer responsible for designing, building, and maintaining data architecture and infrastructure. Develop and deploy data solutions meeting organizational needs.
Lead Data Engineer
Lead Data Engineer building and scaling modern data platforms that power analytics and AI/ML initiatives at Evnek Technologies. Collaborate and mentor while ensuring reliable data infrastructure.
Senior GCP Data Engineer
Senior Consultant in GCP Data Engineering, designing and implementing scalable data solutions at EY. Collaborating with stakeholders and optimizing data pipelines within an agile consulting framework.
Product Owner – Data Platform
Product Owner at Localiza&Co focusing on the evolution of data platforms aligned with company strategies. Collaborating across engineering, data, and business teams.
Senior Data Engineer – PowerBI, Databricks
Senior Data Engineer focusing on Power BI and Azure Databricks for a global reinsurance client. Building centralized reporting platform and contributing to claims data transformation.
Lead Data Engineer, Multiple Positions Available
Lead Data Engineer responsible for designing and implementing data movement solutions for Provident Bank. Collaborating with teams to ensure data warehouse efficiency and quality.
Software Engineer, Data Platform
Software Engineer needed to enhance image capture and analysis platform for recycling tech startup. Create software to sort recyclables and improve sustainability efforts.
Senior Data Engineer
Senior Data Engineer at ZigZag developing data platforms and enabling analytics. Focus on Snowflake architecture and collaboration with multiple teams.
Senior Cloud Data Architect
Data Architect at Vertical Relevance shaping customer cloud journeys with AWS solutions. Designing and implementing data solutions, ensuring customer success through strong technical guidance.
Azure API, Data Engineer
Azure API & Data Engineer part of a strategic LATAM integration project. Focused on Cloud Integration, APIs, and Data Engineering with Azure and AWS.
Manager, Data Engineering
Manager, Data Engineering at Skillable responsible for building data pipelines and managing a team. Leading the engineering practices in a fully remote environment.
Data Engineer
Data Engineer developing data solutions for climate and energy at Carrier Global Corporation. Building and maintaining data pipelines and models to enable analytics and reporting.
Data Engineer
Data Engineer responsible for building infrastructure at Lifebit. Bridging biological science and software engineering to power scientific discoveries while ensuring compliance.
Senior Data Engineer – Microsoft Fabric
Senior Microsoft Data Engineer designing and delivering data solutions for LATAM company. Leading technical implementations and enabling data teams with high-quality data in a remote environment.
Data Engineer
Data Engineer at IRIUM working on banking projects and developing Databricks solutions in fully remote mode. Looking for someone with extensive data engineering experience and Python skills.
Principal Data Architect
Principal Data Architect at HostPapa focusing on shaping data architecture and insights for CloudBlue’s platform. Collaborating across teams to build AI-driven solutions in a cloud-native environment.
Service Manager – DataOps MDG
Service Manager overseeing IT operational services and ensuring service performance at Tec2Cloud. Leading process design and automation initiatives aligning with IT governance in a collaborative setting.
Junior Data Engineer
Jr. Data Engineer at Cutsforth LLC shaping the future of data use in power generation industry through operational data insights. Join our innovative team to design scalable data pipelines.
Senior Business Data Engineer
Senior Business Data Engineer designing, deploying, and maintaining data analytics pipeline for business insights. Collaborating with senior leadership and data teams to improve metrics and processes.