Data Platform Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Data Platform Engineer building PostgreSQL-to-Azure CDC pipelines for Software Mind’s global technology clients. Operating Kafka, Debezium and Kubernetes data infrastructure in production.
Responsibilities:
- Build and operate change-data-capture pipelines from PostgreSQL into Azure using Kafka Connect and Debezium
- Configure, deploy and scale connectors end to end, including connector setup, task management, offsets, schema history and snapshot strategy
- Run pipelines as stateful workloads on Kubernetes (AKS), covering configuration, secrets, networking and resource tuning
- Monitor and troubleshoot the platform in production, including connector failures, task rebalances, restarts, throughput, backpressure, message-size limits, retries and recovery
- Automate the platform in Python through configuration-driven onboarding, pipeline orchestration, monitoring and alerting, recovery workflows and automated testing
- Integrate CDC streams with Azure Event Hubs, ADLS, Azure PostgreSQL, ADF and Databricks
- Manage platform infrastructure as code so environments are reproducible and changes are reviewable
- Apply data protection requirements to sensitive data flowing through pipelines, including masking, hashing, access control and retention
Requirements:
- Solid commercial experience as a data or platform engineer, with hands-on work on streaming or CDC pipelines rather than batch reporting alone
- Practical Kafka knowledge, including topics, partitions, offsets, consumer groups and delivery semantics
- At least one Kafka Connect deployment run independently
- Strong SQL and PostgreSQL skills
- Working knowledge of WAL, logical replication, replication slots and replication lag
- Working understanding of CDC concepts: initial snapshots, inserts, updates and deletes, event ordering, at-least-once delivery, and schema evolution
- Confident Python skills for automation and tooling
- Hands-on experience with Azure data services, such as Event Hubs, ADLS or Azure PostgreSQL
- Comfortable working with Kubernetes as a user, including deploying workloads, handling configuration and secrets, reading logs and debugging failing pods
- Ability to debug running pipelines from metrics and logs
- Production experience with Debezium specifically
- Experience operating stateful workloads on AKS, including StatefulSets, stable worker identity and resource tuning under load
- Infrastructure-as-code and CI/CD experience for data platform components, using Terraform, Bicep or similar
- Hands-on work with Databricks and ADF at production scale
- Experience implementing data protection controls for sensitive data, including masking, hashing, access control and retention policies
Benefits:
- Flexible employment and remote work
- International projects with leading global clients
- International business trips
- Non-corporate atmosphere
- Language classes
- Internal & external training
- Private healthcare and insurance
- Multisport card
- Well-being initiatives
Similar Jobs
Data Engineer
Data Engineer criando soluções de dados em nuvem para empresa brasileira de tecnologia e inovação. Gerenciando Databricks, AWS/Azure, data lakes, data warehouses e pipelines ETL/ELT.
Data Engineer – International Client
Data Engineer building scalable Databricks data products for The White Team’s technology consulting clients. Developing dbt, SQL, Jinja pipelines and modern data architecture remotely across the EU.
Senior Big Data Engineer – AdTech Cybersecurity
Senior Big Data Engineer optimizing Snowflake, SQL, and Java services for a cybersecurity AdTech platform. Improving real-time fraud-risk assessment scalability, performance, and cost efficiency.
Data Platform Engineer
Data Platform Engineer building secure, observable infrastructure for B2Spin’s social casino and gaming products. Managing IaC, IAM, CI/CD, monitoring, cost governance, and disaster recovery.
Data Engineer – UniData, UniVerse
UniData/UniVerse Data Engineer building commission and incentive-processing systems for direct-selling and MLM businesses. Optimizing hierarchies, payouts, integrations, and high-volume transactional data.
Senior Data Engineer
Senior Data Engineer building trusted Snowflake data products for Granum’s landscaper and arborist software ecosystem. Operating governed pipelines that power analytics, AI, and operational decisions.
Ingeniero de Datos – Spark, Kafka, AWS
Ingeniero de Datos diseñando pipelines batch y streaming con Spark, Kafka y AWS. Optimización de datos a gran escala para una multinacional de consultoría IT.
Senior Data Engineer – Microsoft Fabric
Data Engineer building Microsoft Fabric pipelines, Lakehouses, and curated datasets. Supporting analytics, reporting, and AI use cases for Expleo, a global engineering and consulting provider.
Senior Data Engineer
Engenheiro de Dados na DOMVS iT construindo e mantendo pipelines ETL/ELT escaláveis. Desenvolvendo soluções com AWS, Python, SQL e bases de dados.
Data Architect Lead
Data Architect Lead modernizing VA supply-chain data and systems for SAIC, a mission technology integrator. Building cloud integrations, AI/ML analytics, and real-time inventory dashboards.
Data Engineer III
Senior Data Engineer building high-volume data pipelines for Ookla, a global connectivity intelligence company. Supporting data scientists with Spark, Python, SQL, and modern lakehouse technologies.
Data Engineer II
Data Engineer building high-volume pipelines for Ookla, a connectivity intelligence company. Supporting data scientists and analysts with accurate, scalable internet performance data.
Director, Data Engineering – Google Cloud Platform
Director leading TTEC Digital’s Google Cloud data engineering practice. Building enterprise data platforms, GenAI infrastructure, and governance solutions for client transformations.
Senior Data Architect
Senior Data Architect building Snowflake and dbt data foundations for LG Energy Solution Vertech’s energy storage business. Enabling trusted analytics, reporting, governance, and AI across enterprise systems.
Presales Data Architect – Modern Data Stack, AI
Presales Data Architect diseñando arquitecturas Modern Data Stack y soluciones de IA para Reclut, marketplace de reclutamiento con IA. Liderando preventa, PoCs y assessments de arquitectura.
Staff Clinical Informatics Data Architect
Staff Clinical Informatics Data Architect leading FHIR, OMOP, and healthcare data integration for Verily Health's AI-enabled precision-health platform. Overseeing pipelines, standards, and data quality.
Product Manager, Person Data Platform
Product Manager owning Signify Health’s healthcare person-data ingestion platform roadmap. Driving reliable client data intake, platform stability, and HIPAA-compliant care-delivery operations.
Lead Engineer, Data Platform
Lead Engineer building secure, scalable hybrid data platforms for a global investment management firm. Leading architecture, automation, reliability, and enterprise data engineering standards.
Senior Software Engineer – Data Platform
Senior Software Engineer building LightBox's AI platform and developer tooling. Connecting commercial real estate data, AI models, and production workflow applications.
Forward Deployed AI Engineer – Data Intelligence
Forward Deployed AI Engineer building Claude-powered enterprise data and RAG solutions for NewRocket’s ServiceNow clients. Integrating governed data, AI workflows, and enterprise systems for secure operational outcomes.