Senior Data Engineer – AWS, RAG Pipelines
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior Data Engineer designing and operating cloud data infrastructures for AI initiatives. Building data lakes on AWS and real-time pipelines for RAG systems.
Responsibilities:
- Design and operate the cloud data infrastructure powering AI initiatives.
- Architect production-scale data lakes on AWS.
- Build real-time ingestion and observability pipelines.
- Own the vector search and embedding layers that feed RAG systems and autonomous agents.
Requirements:
- Overall Experience: 7+ years in Data Engineering, Distributed Systems, or Data Architecture
- AWS & Infrastructure: 4+ years architecting production-scale data lakes, storage tiers, and event streaming
- AI/LLM Pipelines: 2+ years building RAG systems, managing embeddings, and orchestrating foundational models
- Proficiency in AWS Data Lake Architecture & Storage
- Proficiency in Real-Time Observability & Log Analytics
- Proficiency in Elasticsearch & OpenSearch Optimization, Vectorization, Embeddings
- Proficiency in Amazon Bedrock & Generative AI Pipelines
- Proficiency in Software Engineering & API Ingestion
- Production-level proficiency in one or more of: C# (.NET Core), Java, Python, or Node.js
- AWS S3 partitioning strategies, lifecycle policies, and columnar formats (Parquet, Iceberg)
- AWS Glue Data Catalog and Lake Formation for multi-tenant, fine-grained access control
- Query optimization over petabyte-scale datasets using Amazon Athena and Redshift Spectrum
- Distributed oTel collector configuration for log, trace, and metrics capture and routing into S3
- High-volume streaming of system logs, Datadog captures, and raw server events into S3
- Real-time CDC from PostgreSQL using Debezium or AWS DMS
- Amazon OpenSearch clusters with simultaneous lexical and high-dimensional vector search
- OpenSearch index lifecycle management, sharding strategies, and dynamic mappings at scale
- Amazon Bedrock foundational model APIs (Claude, Titan) for data enrichment, classification, and semantic parsing
- Knowledge Bases for Amazon Bedrock for automatic chunking, metadata extraction, and vector index syncs from S3
- ETL/ELT pipelines ingesting unstructured event data from SaaS APIs (e.g., Pendo, Hotjar, Google Analytics)
- MCP server development to expose data lake context and utilities to AI agents
Benefits:
- Remote work.
- 13 floating holiday.
- 15 vacation days per year completed.
- Good working environment.
Similar Jobs
Data Architect
Advisor Data Architect leveraging technology to improve health outcomes managing Medicaid and Major Health Care Payer data projects. Collaborating with clients to ensure data integrity and governance.
Business Data Engineer
Business Data Engineer in a flexible role at Gainwell, focusing on data solutions for healthcare improvement. Engage in data analysis, collaboration, and innovative problem-solving to enhance healthcare outcomes.
Senior Data Architect, Data Engineer – AI & Data Platforms
Senior Data Architect / Data Engineer designing scalable enterprise data solutions leveraging MDM and AI. Collaborating with teams to implement data strategies across the organization.
Junior Data Engineer
Junior Data Engineer at iA Financial Group supporting data management team in maintaining cloud data warehouse. Responsibilities include ETL/ELT pipeline development, data quality assurance, and collaboration across teams.
Senior GCP Data Engineer
Senior Data Engineer designing and maintaining efficient data pipelines on GCP. Collaborating with teams to implement scalable data solutions that meet business needs.
Data Engineer – Contractor
Data Engineer optimizing AI workflows and building data pipelines for BiOptimizers. Collaborating on system architecture and ensuring data quality in a remote-first environment.
Senior Product Manager, Health Systems Data Platforms
Senior Product Manager leading the vision and strategy for healthcare data platforms at Natera. Focused on integrating genomic and clinical data to enhance patient outcomes.
Data Engineer
Data Engineer at ICF focusing on legacy system modernization and data migration into the cloud. Drive analytics and operational efficiency through MuleSoft integrations and data quality processes.
Lead Data Engineer
Lead Data Engineer at Egen focusing on building and optimizing cloud-native data platforms on Google Cloud. Mentoring engineers and designing sustainable data architectures with a data-first mindset.
Senior Data Engineer
Senior Data Engineer at DTN responsible for designing and implementing data solutions with cloud architecture. Mentoring teams and ensuring data governance in a dynamic environment.
Data Engineer
Data Engineer responsible for building data pipelines and configuring customer instances for B2B sales intelligence. Collaborating with Customer Success to deliver data solutions.
Data Engineer Manager
Lead Data Engineering Manager at Blend, focused on designing scalable solutions for Journey Analytics. Overseeing data platforms and leading a team of engineers to deliver high-quality datasets.
Data Engineer Manager
Data Engineer Manager leading a team for Journey Analytics initiatives at Blend, an AI services provider. Focus on scalable data solutions and data engineering best practices.
Data Migration Analyst
Data Migration Analyst at NikoHealth overseeing transitions from legacy platforms into SaaS solutions. Ensure data accuracy and provide customer support during the migration process.
Senior Data Engineer – Databricks, AWS
Senior Databricks Data Engineer specializing in AWS cloud-native data solutions. Supporting federal client data platform modernization initiatives with expertise in data pipelines and governance.
Freelance Data Engineer
Data Engineer at Incubeta responsible for designing, building, and maintaining data processing systems to ensure data integrity and optimize pipelines. Mentors junior members and delivers data solutions effectively.
Senior Data Engineer
Senior Data Engineer delivering modern data solutions at Carrier, focused on data pipelines and lakehouse architectures. Collaborating with cross-functional teams to support analytics and reporting.
Lead Product Manager, Data Platform
Lead Product Manager at Alkami driving the data platform strategy for the fintech sector. Collaborate cross-functionally to enhance data products in a remote-first environment.
Data Warehouse Developer
Data Warehouse Developer assisting in the analytics solutions for the University of Wisconsin-Platteville. Collaborating in the design, build, testing, and deployment of data warehousing.
Senior Data Engineer
Senior Data Engineer building resilient data pipelines for Verantos's high-accuracy RWE generation platform. Leading data architecture and collaborating with cross-functional teams in a remote US-based role.