Senior Software Engineer, Ingestion Team
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior Software Engineer building AI processing pipelines for a leading knowledge management platform at Pryon. Designing and implementing robust data ingestion architectures and collaborating with cross-functional teams.
Responsibilities:
- Design and build pipeline stages for our modern ingestion architecture - from document intake through embedding generation and index writing
- Contribute to the design of next-generation pipeline architecture as the system evolves
- Improve system stability and scale: identify bottlenecks, reduce failure rates, and build observability into every stage
- Work with workflow orchestration tools to manage complex, multi-step document processing with retry logic, error handling, and state management
- Handle the realities of document diversity: PDFs, HTML, Office formats, images, structured and semi-structured data - all flowing through the same pipeline
- Collaborate with the Connectors team (upstream) and Retrieval team (downstream) to ensure data flows cleanly across system boundaries
- Participate in the ongoing migration from legacy systems, balancing new development with operational stability
Requirements:
- 5+ years of software engineering experience, with meaningful time on data processing pipelines, ETL systems, or similar infrastructure
- Strong proficiency in Python and/or Go
- Experience with workflow orchestration tools — Temporal, Airflow, Prefect, Step Functions, or similar
- Understanding of distributed systems patterns: queues, workers, backpressure, idempotency, retry strategies
- Hands-on experience with Kubernetes, Docker, Terraform, and Helm
- Familiarity with message brokers and event streaming (Kafka, RabbitMQ, SQS, or similar)
- Comfort working across cloud providers (AWS, Azure, GCP)
Benefits:
- Remote first organization
- 100% Company paid Health/Dental/Vision benefits for you and your dependents
- Life Insurance, Short-term and Long-term Disability
- 401k
- Unlimited PTO
Similar Jobs
Senior Software Engineer II – App Observability
Senior Software Engineer II on Honeycomb's App Observability Team. Delivering core observability capabilities and building APIs and backend services.
Senior Software Engineer
Senior Software Engineer shaping healthcare automation solutions within the PAT Assist team at Qventus. Developing impactful, scalable systems to enhance pre-admission testing workflows.
Lead Software Engineer, AI
Lead Software Engineer for Thomson Reuters, developing AI-powered systems enhancing accounting workflows. Collaborate in a dynamic environment with experts in AI and software development.
Senior System Software Engineer, OpenBMC
Senior System Software Engineer responsible for implementing firmware architecture using OpenBMC for NVIDIA Server platforms. Collaborating with hardware teams and driving product life cycles with QA teams.
Desenvolvedor Full Stack – Pleno
Full Stack Developer joining Mollica IT to work on innovative projects. Utilize Node.js and AI tools in a fully remote setting.
Senior Software Engineer – Fabric Networking, GPU
Senior Software Engineer responsible for building high-performance communication software for GPU Fabric Networking. Collaborating across teams to develop and maintain scalable networking solutions.
Software Engineer, Automated Marketing
Full-stack engineer developing and enhancing tools for marketing automation at Warner Music Group. Collaborating with marketing and product teams on innovative solutions using Python and React.
Full-Stack Developer, Deployment & Operations
Hands-on Deployment & Application Engineer managing the migration of a web platform for a visual communications client. Significant technical ownership in a fast-moving, product-focused organization.
Senior Software Engineer, CV Applications
Senior Software Engineer delivering backend infrastructure and computer vision features for a youth sports platform. Collaborating with cross-functional teams to create innovative user experiences.
Senior Software Engineer – Front End Architecture, React Native
Senior Software Engineer developing and maintaining a React Native mobile app for B2B operations. Collaborating with teams to enhance cross-platform performance and user experience.
Senior Full Stack Engineer, Next.js
Senior Next.js Full-Stack Engineer at Ruby Labs working on scalable D2C platforms. Responsible for architecture, reliability, and mentoring in a rapid growth environment.
Software Engineer III
Software Engineer III role supporting Agile Scrum team at Clayton Homes. Involved in software development process from planning to delivery, ensuring quality and reliability of applications.
Full Stack Developer, AI (Mid-Level)
Full Stack Developer with AI focus for SaaS solutions, contributing to engineering and mentorship.
Principal AI Product Engineer
Designing scalable AI capabilities and components for healthcare improvement. Collaborating with product teams, and implementing advanced technologies using the Databricks platform.
Tech Lead – Full Stack
Tech Lead driving migration from legacy Golang to modern Python architecture for healthcare data. Focused on high-quality medical imaging ensuring scalability and compliance standards.
Tech Lead – Arquiteto Salesforce SR
Tech Lead managing Salesforce Sales Cloud and Service Cloud solutions at Grupo SysMap. Leading technical architecture decisions and mentoring developers in a collaborative environment.
Solutions Architect – Azure PaaS, Software Architecture
Solutions Architect specializing in Azure PaaS and software architecture at Grupo SysMap. Responsible for designing solutions and guiding modernization efforts.
Software Engineer Apprenticeship Program – Accelerated Track
Software Engineer Apprentice at Flatiron School. Attend bootcamp and gain hands-on industry experience in AI and data science over 14 months.
Senior Fullstack Engineer – AI
Senior Fullstack Engineer designing and delivering AI solutions for Bloom Growth's platform. Collaborating closely with product teams to enhance customer experiences through innovative technologies.
Databricks Tech Lead
Tech Lead planning and executing data engineering releases at Capgemini Engineering. Collaborating with development pods, managing project timelines, and executing deployment plans in production environments.