Principal Engineer – Data Retrieval
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Principal Engineer architecting web-scale crawling, ingestion, and retrieval infrastructure for NVIDIA’s foundation AI models. Leading distributed systems design, production scaling, and high-quality pretraining data delivery.
Responsibilities:
- Architect and scale data acquisition infrastructure powering NVIDIA’s foundation models
- Build systems that discover valuable sources, crawl the open web at scale, and deliver reliable, high-quality data to training pipelines
- Own architecture of a unified platform covering source discovery, URL indexing, crawl frontiers, predictive ranking, scheduling, distributed fetching, and ingestion
- Design scalable frontier storage and queueing systems with high-throughput writes, efficient lookups, deduplication, and quality-, diversity-, and freshness-based scheduling
- Build a continuous discovery loop that extracts links and feeds new candidates into the crawl frontier
- Develop a shared crawl execution layer that fetches reliably and politely at scale, including dynamic websites
- Build extensible pipelines for open-web, licensed, and partner content, including text, PDFs, code repositories, and multimodal assets
- Define stable interfaces for source connectors
- Partner with parsing and curation teams to preserve valuable code, math, and structured content and improve accepted data yield
- Implement provenance and policy enforcement with legal, privacy, security, and data governance partners
- Translate publisher preferences, opt-outs, licensing terms, and use restrictions into auditable downstream controls
- Own production readiness, including capacity planning, observability, fault tolerance, recovery, and cost efficiency
- Instrument the pipeline end to end and set service objectives for throughput, freshness, reliability, and training cutoffs
- Evaluate internal crawls and external datasets through comparable pilots with research and product teams
- Deliver reproducible data releases tied to source coverage and processing quality in controlled training experiments
- Coordinate cross-team dependencies and balance immediate data delivery with long-term platform development
- Lead architecture reviews and critical implementation work
- Mentor engineers and help build the team
Requirements:
- BS, MS, or PhD in Computer Science, Computer Engineering, or a related technical field, or equivalent experience
- 15+ years of relevant experience in software engineering
- Experience designing and operating large-scale distributed systems
- Experience setting technical direction and delivering complex infrastructure across multiple teams
- Deep experience with web crawling or closely related discovery, indexing, or ingestion infrastructure
- Experience with distributed scheduling, stateful services, storage, and high-throughput data processing
- Strong hands-on engineering and system design skills
- Ability to move between architecture, implementation, performance analysis, and production debugging
- Experience building batch or streaming pipelines
- Knowledge of consistency, retries, duplicate processing, backpressure, and recovery
- Track record of improving throughput, latency, resource efficiency, or operational reliability
- Experience delivering data platforms for research, ML, search, or other demanding technical users
- Ability to translate evolving requirements into clear interfaces and measurable engineering outcomes
- Strong technical communication and leadership through influence across engineering, product, research, and governance teams
- Experience architecting a web-wide crawl platform, scaling crawl frontier storage, or consolidating multiple acquisition systems into shared infrastructure
- Direct experience supplying foundation-model pretraining data, including targeted discovery, quality-aware crawl ranking, and acquisition of code or technical content
- Experience implementing publisher consent, training-use opt-outs, dataset licensing controls, or provenance infrastructure at scale
- Background with browser-based or agentic data collection, programmable extraction, and sandboxed environments for tool-use data or post-training evaluation
- Experience onboarding licensed or multimodal datasets, building reusable partner integrations, or delivering streaming feeds with low latency for retrieval and grounding
Benefits:
- Equity
- Benefits
Similar Jobs
Data Engineering Lead
Data Engineering Lead guiding engineers and consolidating Oxford Global Resources’ Azure and Microsoft Fabric data platform. Owning architecture, ETL reliability, governance, and delivery.
IT Data Science Principal, Data Warehouse
Leading enterprise data warehouse, analytics, and AI capabilities for Cornerstone Building Brands, a North American building-products manufacturer. Driving roadmaps, governance, modernization, and multidisciplinary delivery.
Data Engineer
Data Engineer building reliable pipelines for Branch’s fintech platform, which accelerates payments and provides financial services. Using Python, SQL, Airflow, Snowflake, DBT, and GCP.
Data Engineer
Data Engineer building reliable pipelines and transformations for Definitive Healthcare’s healthcare data products. Supporting Claims and Consumer domains with Python, SQL, Spark, and cloud platforms.
Senior Data Platform Engineer
Senior Data Platform Engineer building scalable data lakes, ingestion pipelines, and Kubernetes infrastructure. Supporting analytics, product applications, AI workloads, and large-scale model execution.
Medical Receptionist, Data Migration Specialist
Remote medical receptionist and data migration specialist transferring patient records from Elation Health to Dakota Health. Supporting U.S. practice scheduling, patient calls, and administrative workflows.
Data Engineer
Data Engineer building Azure Databricks pipelines for credit-card data at CGI, a global IT and business consulting firm. Modernizing financial data platforms and controls.
Working Student – Data Engineering
Working Student building Python and SQL data pipelines for AgentixLake’s cloud data and AI engineering projects. Supporting data quality, lakehouse platforms, testing, and monitoring.
Data Engineer – Temporary
Data Engineer building reliable Zuora-to-GCP ETL pipelines for Applaudo’s AI-powered solutions. Maintaining integrations, data quality, monitoring, and warehouse validation in an async distributed team.
WEKA Data Platform Administrator
WEKA Data Platform Administrator operating NeuralMesh enterprise AI data infrastructure for GPU-cloud customers. Managing storage, Kubernetes, networking, automation, and customer workload performance.
Analytics Engineer – Data Platform
Analytics Engineer building Snowflake and dbt models for Aline’s senior-living resident lifecycle platform. Creating governed data foundations for BI reporting and AI-driven care outcomes.
Senior Data Engineer – Mid Shift
Data Engineer building AWS and Snowflake platforms for Shore360, an Australian-owned Philippine staffing and administrative support firm. Developing reliable pipelines, APIs, and data quality controls.
Senior Software Engineer, Data Infrastructure – RDBMS
Senior engineer operating petabyte-scale PostgreSQL infrastructure for TRM Labs’ AI-powered crime-intelligence platforms. Automating database operations, optimizing performance, and managing CDC pipelines remotely across LATAM.
Data Architect
Databricks Data Architect designing secure Lakehouse architectures and AI-ready data platforms for Devoteam clients. Modernizing legacy systems across cloud and multi-cloud environments.
Senior Data Engineer
Senior Data Engineer building auditable GCP data pipelines for DysrupIT’s financial billing and invoicing platform. Ensuring accurate reconciliation, lineage, and trusted financial data.
Senior Software Engineer, Data Acquisition
Senior software engineer building distributed crawling and ingestion systems for People Data Labs, a provider of people and company data. Improving scalable backend services and data products.
IT Architect, Data Scientist
IT Architect/Data Scientist supporting DLH’s federal health and national-security programs. Leading cloud architecture, data science, interoperability, security engineering, and compliance for SCHARE.
Senior Business Data Engineer
Senior Business Data Engineer building AWS data infrastructure and automated reporting for Nex’s family motion-gaming platform. Improving pipeline reliability and supporting cross-functional business teams.
Senior Data Engineer
Senior Data Engineer building Python, Spark, and Databricks pipelines for Quantiphi’s data and AI solutions. Leading cloud migration, CI/CD automation, and scalable analytics engineering.
Senior Data Engineer – IBM DB2
Senior Data Engineer modernizing IBM DB2 data environments. Building reliable ETL/ELT pipelines and AWS integrations for enterprise applications and payroll systems.