Engenheiro de Dados Sênior – Especialista em Crawlers e Orquestração
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior Data Engineer leading data ingestion using Python and AWS at Cortex. Implementing complex data pipelines and optimizing data flow in cloud environments.
Responsibilities:
- Desenvolver e manter crawlers/bots de alta performance e resilientes para extração de dados em larga escala.
- Projetar e implementar pipelines de dados complexos utilizando Databricks (Spark) para processamento em lote e streaming.
- Garantir a saúde e a dependência dos fluxos de dados através de ferramentas de orquestração avançadas.
- Gerenciar e otimizar recursos dentro do ecossistema AWS, garantindo escalabilidade e custo-benefício.
- Implementar técnicas de tratamento de erros, contorno de bloqueios (proxies, captchas) e validação de qualidade dos dados coletados.
Requirements:
- Domínio profundo de Python (focado em bibliotecas de scraping como Scrapy, Playwright, Selenium ou Beautiful Soup).
- Experiência sólida com Databricks e Apache Spark (PySpark).
- Experiência com serviços como AWS S3, Lambda, Glue, Athena, EC2 e EKS.
- Conhecimento avançado em ferramentas como Airflow, Dagster ou Prefect.
- Experiência com bancos SQL e NoSQL, além de entendimento de Data Lakehouses (Delta Lake).
- Familiaridade com Docker, Kubernetes e pipelines de CI/CD.
- Conhecimento em técnicas de evasão de bot-detection e gestão de redes de Proxy (diferencial).
- Experiência com ferramentas de monitoramento (Prometheus, Grafana ou ELK) (diferencial).
- Contribuições para projetos Open Source ou comunidades de dados (diferencial).
Benefits:
- Vale Refeição e Alimentação;
- Gympass/TotalPass;
- Vittude;
- Auxílio home-office;
- Plano de Saúde e Plano Odontológico (odonto é opcional);
- Auxílio creche (até os 6 anos completos da criança);
- Licença-Maternidade, Paternidade e Adotante Estendidas (#todasasfamíliasimportam);
- Seguro de Vida;
- Day Off de Aniversário (Um dia de folga para tirar no dia ou durante o mês do seu aniversário.);
- Dia da Família (Um dia de folga para mamães e papais tirarem entre os meses de maio e agosto e aproveitar como quiserem.);
- Pausa Mental (Uma semana corrida de folga em dezembro para que descansem e recarreguem as baterias.
Similar Jobs
Power Systems Study Engineer – III
Power Systems Study Engineer III at DLB Associates working on power system studies for mission critical projects. Collaborating within the Engineering Team to optimize performance.
Cloud Operation Engineer
Cloud Operation Engineer responsible for designing, implementing, and maintaining server infrastructure at CSG Systems, Inc. Ensuring stability and security for enterprise systems in both on-premise and cloud environments.
Senior I&C Engineer – Energy
Senior I&C Engineer Energy designing and maintaining systems for nuclear facilities. Delivering technical documentation and ensuring compliance with nuclear safety regulations.
DFT Engineer
Senior DFT Engineer leading end-to-end DFT strategy for complex SoCs at Niobium Microsystems. Overseeing architecture, integration, and validation processes within the team.
External Lighting Engineer
External Lighting/Street Lighting Engineer at Fixatex performing installations and maintenance of street and external lighting across London. Ensuring compliance with safety standards and delivering quality service in social housing and local authority sectors.
Thermal Hydraulics Engineer 3
Thermal Hydraulics Engineer developing thermal systems analysis for NuScale Power’s microreactor technology. Collaborating on modeling, analysis, and software selection within a dynamic team.
Interconnection Engineer
Interconnection Engineer managing engineering items for clean energy solutions at WECTEC Staffing Services. Collaborating with corporate teams and Transmission Owners to ensure interconnection agreements are met.
Sandbox Agentic Engineer
AI Agent Engineer developing LLM-based AI solutions for cybersecurity at CrowdStrike. Leading AI agent development and optimizing performance with modern frameworks and technologies.
Critical Power Field Technician, UPS Engineer
Critical Power Technician maintaining critical power systems in Nashville, TN. Role involves technical support, customer service, and mentoring crew members in UPS systems.
Forward Deployed Engineer
Forward Deployed Engineer at Truelogic Software integrating AI solutions with clients. Collaborating on impactful projects to enhance digital transformations in businesses.
LLM Evaluation Engineer
Develop evaluation systems for AI behavior in enterprise environments using LLMs and real-time data. Collaborate on guardrails and enforcement strategies for AI compliance.
Chief Engineer – Container Vessel
Chief Engineer leading Engine room operations for Columbia Shipmanagement's fleet, ensuring equipment functionality and safety. Requires strong experience with specific engine types and good communication skills.
Technical Engineer
Technical Engineer for the Wendy's engineering team focusing on mobile development and problem solving.
Staff IT/MSP Engineer
Staff IT Engineer responsible for technical health of MSP's client base, designing deployment standards. Bridging business goals with technical execution, reporting to CTO or Director of Engineering.
Customer Delivery Engineer
Customer Delivery Engineer at Tiger Data implementing PostgreSQL projects and optimizing customer experiences with AI tools during migrations and production rollouts.
Senior Identity Access Management Engineer
Senior IAM Engineer improving Identity and Access Management processes at U.S. FinTech. Responsible for automation, documentation and overseeing AWS IAM solutions for security compliance.
Senior Software V&V Engineer
Senior Software V&V Engineer ensuring thorough verification and validation of software in cancer diagnostics. Collaborate with cross-functional teams for bioinformatics pipeline testing and regulatory compliance.
Project Engineer
Project Engineer working on maintenance and development of automation systems. Involves offshore operations and all stages of automation lifecycle support in Rio de Janeiro.
Project Engineer
Project Engineer maintaining automation systems for customer facilities including offshore operations. Engaging in all stages of systems lifecycle from requirements gathering to startup.
Systems Monitoring Engineer
Systems & Monitoring Engineer architecting monitoring stack for Hedera’s DLT ecosystem. Blending NOC practices with Web3 challenges to ensure uptime and compliance.