Senior Software Engineer, Data Acquisition
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior software engineer building distributed crawling and ingestion systems for People Data Labs, a provider of people and company data. Improving scalable backend services and data products.
Responsibilities:
- Contribute to the architecture and improvement of the data acquisition and processing platform, increasing reliability, throughput, and observability
- Use and develop web crawling technologies to capture and catalog data on the internet
- Build, operate, and evolve large-scale distributed systems that collect, process, and deliver data from across the web
- Design and develop backend services for distributed job orchestration, data pipelines, and large-scale asynchronous workloads
- Structure and model captured data, ensuring high quality and consistency across datasets
- Improve the speed, scalability, and fault tolerance of ingestion systems
- Partner with data product and engineering teams to design and implement new data products and improve existing products
- Learn and apply domain-specific knowledge in web crawling and data acquisition with mentorship from experienced teammates
Requirements:
- 7+ years of professional experience building or operating backend or infrastructure systems at scale
- Solid programming experience in Python, Go, Rust, or similar, including async / await, coroutines, or concurrency frameworks
- Strong grasp of software architecture and backend fundamentals, including concurrency, scalability, and fault tolerance
- Solid understanding of browser rendering pipeline and web application architecture, including authentication, cookies, and HTTP request/response
- Familiarity with network architecture and debugging, including HTTP, DNS, proxies, and packet capture and analysis
- Solid understanding of distributed systems concepts: parallelism, asynchronous programming, backpressure, and message-driven design
- Experience designing or maintaining resilient data ingestion, API integration, or ETL systems
- Proficiency with Linux / Unix command-line tools and system resource management
- Familiarity with message queues, orchestration, and distributed task systems such as Kafka, SQS, and Airflow
- Experience evaluating and monitoring data quality, ensuring consistency, completeness, and reliability across releases
- Ability to work independently in a fast-paced, remote-first environment and collaborate asynchronously
- Ability to communicate clearly and thoughtfully in writing
- Ability to write and maintain technical design documents, including pipeline design, schema design, and data flow diagrams
- Ability to scope and break down complex projects into deliverable milestones and communicate progress, risks, and blockers
- Degree in a quantitative field such as computer science, mathematics, or engineering (nice to have)
- Experience as a Red Teamer (nice to have)
- Experience with large-scale data ingestion, crawling, or indexing systems (nice to have)
- Experience with Apache Spark, Databricks, or other distributed data platforms (nice to have)
- Experience with streaming data systems such as Kafka, Pub/Sub, and Spark Streaming (nice to have)
- Proficiency with SQL and data warehousing such as Snowflake, Redshift, or BigQuery (nice to have)
- Experience with cloud platforms, preferably AWS (nice to have)
- Understanding of modern data storage and design patterns such as Parquet, Delta Lake, partitioning, and incremental updates (nice to have)
- Experience building and maintaining data pipelines on modern big-data or cloud platforms (nice to have)
Benefits:
- Stock
- Competitive Salaries
- Unlimited paid time off
- Medical, dental, & vision insurance
- Health, fitness, and office stipends
- The permanent ability to work wherever and however you want
Similar Jobs
Data Architect
Databricks Data Architect designing secure Lakehouse architectures and AI-ready data platforms for Devoteam clients. Modernizing legacy systems across cloud and multi-cloud environments.
Senior Data Engineer
Senior Data Engineer building auditable GCP data pipelines for DysrupIT’s financial billing and invoicing platform. Ensuring accurate reconciliation, lineage, and trusted financial data.
IT Architect, Data Scientist
IT Architect/Data Scientist supporting DLH’s federal health and national-security programs. Leading cloud architecture, data science, interoperability, security engineering, and compliance for SCHARE.
Senior Business Data Engineer
Senior Business Data Engineer building AWS data infrastructure and automated reporting for Nex’s family motion-gaming platform. Improving pipeline reliability and supporting cross-functional business teams.
Senior Data Engineer
Senior Data Engineer building Python, Spark, and Databricks pipelines for Quantiphi’s data and AI solutions. Leading cloud migration, CI/CD automation, and scalable analytics engineering.
Senior Data Engineer – IBM DB2
Senior Data Engineer modernizing IBM DB2 data environments. Building reliable ETL/ELT pipelines and AWS integrations for enterprise applications and payroll systems.
Lead Data Architect
Lead Data Architect owning enterprise data architecture for C5MI Insight’s federal SAP S/4HANA modernization program. Leading data governance, migration, analytics architecture, and master data teams.
FBS Data Engineer, SQL, Power BI
Data Engineer supporting Farmers’ P&C insurance analytics and financial loss reporting solutions. Translating business data needs into requirements, integrations, dashboards, and validated reporting.
Data Engineer – Insurance
Data Engineer building analytics solutions, data pipelines, and enterprise integrations for Farmers Insurance. Improving data quality, reporting, dashboards, and regulatory-compliant business outcomes.
Power BI Data Engineer
Power BI Data Engineer building Azure data pipelines, dimensional models, and executive dashboards for Legence. Transforming complex data into actionable business intelligence across the full BI lifecycle.
Azure Data Engineer
Azure Data Engineer building Azure and Microsoft Fabric data solutions for Quisitive, a global Microsoft cloud partner. Delivering pipelines, integrations, and analytics projects for clients.
Senior Data Engineer
Senior Data Engineer building multi-region data infrastructure for Horizon3’s NodeZero autonomous cybersecurity platform. Supporting Federal workloads, warehouses, pipelines, orchestration, and BI reporting.
Data Engineer, Snowflake
Data Engineer Snowflake diseñando pipelines cloud y transformaciones ELT/ETL para Logicalis Spain. Participación en proyectos de ingeniería de datos, IA y analítica avanzada.
Data Engineer
Data Engineer building Python and AWS data pipelines for 540’s U.S. Army financial-system modernization initiative. Improving secure, scalable data transformation and governance.
Senior Manager, Data Engineering
Data engineering leader building Databricks and Azure platforms for Cushman & Wakefield, a global commercial real estate services firm. Managing engineers, platform delivery, governance, and stakeholder outcomes.
Data Architect
Data Architect designing Azure-integrated data platforms for Tieto’s agricultural analytics projects. Building scalable datasets, ETL pipelines, warehouses, and lakes.
Data Engineer
Data Engineer building scalable cloud pipelines, models, and enterprise data products for a major U.S. insurer. Supporting analytics, reporting, governance, and operational data capabilities.
Senior Data Engineer
Engenheiro de Dados SR desenvolvendo pipelines, Data Lakes e governança para o Grupo SysMap. Empresa brasileira de tecnologia acelerando a transformação digital de organizações.
Data Engineer
Data Platform Engineer owning cloud infrastructure, Terraform, CI/CD, and observability for Seer Interactive’s digital marketing consultancy. Automating analytics and AI platform operations.
Data Engineering Lead
Data Engineering Lead architecting Seer Interactive’s GCP data platform and agentic engineering workflows. Enabling governed analytics and AI products for digital marketing clients.