Director of Software Engineering – Node.js, Web Scraping Expert
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Director of Software Engineering leading web scraping systems development with Node.js. Overseeing engineering team and optimizing large-scale data extraction processes.
Responsibilities:
- Architect, develop, and maintain scalable and distributed web scraping systems using Node.js.
- Design and implement data extraction pipelines to process large volumes of structured and unstructured data.
- Develop solutions to bypass anti-bot mechanisms, including CAPTCHA handling, session management, fingerprinting, and IP rotation.
- Optimize scraping processes for performance, reliability, and efficiency while managing proxy services (residential, datacenter, rotating).
- Oversee data storage and processing strategies, ensuring high availability and consistency.
- Collaborate with Product, DevOps, and Data Science teams to integrate extracted data into analytics and business applications.
- Implement best practices for microservices, API integrations, and real-time data streaming.
- Lead the transition to cloud-native, containerized, and serverless architectures for web scraping.
- Ensure compliance with legal and ethical standards (robots.txt, GDPR, CCPA, etc.).
- Optimize cloud resources (AWS, GCP, or Azure) to support high-throughput scraping.
- Manage real-time monitoring and alerting systems to detect scraping failures, IP bans, or performance bottlenecks.
- Work closely with DevOps teams to optimize CI/CD pipelines, automated deployments, and system scalability.
- Lead, mentor, and grow a high-performance engineering team.
- Define and execute the technology roadmap, aligning with business objectives.
- Foster a culture of continuous learning, collaboration, and innovation.
- Implement agile development methodologies (Scrum, Kanban) to optimize project execution.
- Ensure code quality, security, and best practices across all engineering efforts.
Requirements:
- 10+ years of experience in software engineering, with at least 5+ years in web scraping and large-scale data extraction.
- Strong hands-on expertise in Node.js, Puppeteer, Playwright, Cheerio, Selenium, and headless browser automation.
- Extensive experience in handling CAPTCHAs, IP rotation, session management, and anti-bot evasion techniques.
- Deep knowledge of proxy management (residential, datacenter, rotating, and VPNs).
- Experience with NoSQL/SQL databases (MongoDB, PostgreSQL, Redis, Elasticsearch, etc.).
- Familiarity with data processing frameworks (Kafka, RabbitMQ, Spark, Airflow, etc.).
- Strong experience with CI/CD, containerization (Docker, Kubernetes), and cloud deployment (AWS/GCP/Azure).
Similar Jobs
Lead Technical Architect
Lead Technical Architect overseeing Salesforce initiatives aligned with business strategies. Collaborating with teams to deliver technical solutions that enhance operations in the company.
Senior Software Engineer – Golang, Python
Software Engineer with experience in back-end systems and data-focused development for fintech company. Collaborating with global team in a cloud-native environment to deliver financial solutions.
Senior Database Engineer
Senior Database Engineer at Sales Hub responsible for database design, implementation, and performance optimization. Collaborating with teams to ensure data integrity and implement security measures.
Senior BackEnd Developer – NodeJS
Sr BackEnd Developer (NodeJS) for full stack development on healthcare platform. Collaborating with stakeholders in a dynamic remote work setting for LATAM.
.Net Developer
.NET Developer supporting DoD Child and Youth Programs project within ICF’s Digital Modernization Division. Collaborating with cross-functional teams to design and develop scalable web applications.
Senior Server Engineer – NBA 2K
Senior Engineer for NBA 2K focusing on server-side development including backend architecture and large-scale systems. Collaborating with engineering teams to optimize online gameplay experiences.
Desenvolvedor Java, Pleno
Java Developer building high-performance solutions for the traditional Brazilian brand Hering. Seeking passionate coders with collaborative spirit in technology team.
Principal Backend Engineer
Principal Backend Engineer leading scalable backend systems development for HARMAN's music discovery platform. Focusing on API design, performance optimization, and technical innovation within music data services.
Technical Architect – Messaging
Technical Architect designing secure and scalable Microsoft 365 identity solutions. Collaborating across teams to enhance email messaging and data protection with innovative infrastructure strategies.
.NET Core Developer
.NET Core Developer maintaining multiple healthcare applications at Gainwell. Contributing to application support, maintenance, and development using .NET and SQL technologies.
Senior Software Engineer, Backend
Senior Software Engineer developing backend solutions for Atticus, a platform connecting people with disabilities to government benefits. Leading architecture and mentoring engineers in a cross-functional team.
Senior Backend Engineer, Context Hub
Senior Backend Engineer designing foundational data infrastructure at Oura, focusing on data transformation and event-driven architectures. Collaborating across diverse product teams for innovative health solutions.
Senior Backend Engineer
Senior Backend Engineer at Ruby Labs developing scalable backend systems for AI product Use.ai. Requires strong expertise in distributed systems and collaboration within remote teams.
Full Stack Developer, Python
Full Stack Python Developer focused on developing dynamic web applications and APIs. Work with modern technologies in a flexible environment, contributing to impactful projects.
Java Developer – Mid/Senior, Python
Java Developer responsible for backend architecture and microservices at Salve.Inno Consulting. Engaging in product decisions and collaborating with multiple engineering teams.
Senior Backend Developer, PHP
Senior Backend Developer contributing to backend solutions that support language access services. Focused on PHP, Symfony, and REST APIs within a multicultural global team.
Junior Backend Developer, Python
Junior Backend Developer working with Python in an automated content production environment at uNaice GmbH. Involves API development and data processing for e-commerce and publishing sectors.
Senior Developer, C#, .NET, SQL
Senior Developer at Fresche Solutions focusing on backend solutions with C# and SQL for data applications. Collaborating with clients and optimizing database performance in a flexible remote environment.
Senior Developer – C#, .NET, SQL
Senior Developer at Fresche Solutions specializing in C#, .NET, and SQL. Designing, building, and optimizing data-driven application solutions with complex queries and backend systems.
Senior Backend Java – Event-Driven Architecture
Senior Backend Java position focusing on event-driven architecture for FlyingHub, a drone management platform. Responsible for building robust backend systems and ensuring modular design.