Data Architect, Databricks
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Data Architect migrating Databricks workloads to AWS data platforms for CI&T’s enterprise AI transformation clients. Building scalable pipelines, governance, integrations, and secure analytics infrastructure.
Responsibilities:
- Build, optimize, and maintain scalable data pipelines and ETL/ELT processes for analytics and business intelligence
- Implement data architectures, including data lakes, warehouses, and integration layers
- Implement data quality checks, validation processes, and monitoring systems
- Develop and maintain integrations between data sources, third-party tools, and analytics platforms
- Collaborate with Data & Analytics Managers and business stakeholders to understand requirements and deliver technical solutions
- Monitor and optimize data pipeline performance, query efficiency, and storage costs
- Create and maintain technical documentation for data architectures, pipelines, and processes
- Implement security best practices and ensure compliance with data privacy regulations
- Leverage AI tools and automation to enhance data engineering workflows and productivity
Requirements:
- Hands-on Databricks migration with the databricks/migrate tool, including notebooks, jobs, clusters, secrets, and ACLs
- Unity Catalog to Glue Catalog migration experience
- Notebook and job migration, including notebooks, DAG pipelines, and DLT pipelines
- Storage path refactoring from /mnt/ and DBFS references to S3-backed locations
- Data migration execution from Delta Lake format to Iceberg format on S3, including Delta tables, raw files, and incremental sync
- Auto Loader / DLT checkpoint re-bootstrap and event_id deduplication
- Validation using row counts, checksums, sample queries, and ETL runtime comparison
- Recreate secret scopes in AWS Secrets Manager
- Experience with AWS S3, Glue, Secrets Manager, IAM basics, Athena, Lake Formation, and EMR
- Proficiency in Spark SQL, PySpark, Java, Python, and Scala
- Agile pod delivery, including daily ticket updates, standups, and code review
- Databricks Certified Consultant certification and DBR to EMR on AWS migration experience are nice to have
Benefits:
- Premium Healthcare
- Meal voucher
- Maternity and Parental leaves
- Mobile services subsidy
- Sick pay-Life insurance
- CI&T University
- Colombian Holidays
- Paid Vacations
- And many others
- Diverse, inclusive, and safe work environment
Similar Jobs
Staff Data Engineer
Staff Data Engineer building scalable data systems for Oviva’s AI-powered chronic care platform. Designing pipelines, data products, and quality frameworks across European healthcare operations.
Senior Data Migration Specialist
Senior Data Migration Specialist leading secure EMR migrations for WELLSTAR’s Canadian healthcare platforms. Improving data tools, validation, governance, and go-live readiness.
Senior Data Engineer
Senior Data Engineer building cloud-native Databricks and AWS data platforms for a commercial laser fusion company. Designing real-time pipelines, observability, and AI-enabled engineering workflows.
Associate Data Engineer
Associate Data Engineer building Azure, Synapse, and Microsoft Fabric data solutions for Baker Tilly’s mid-market and enterprise clients. Developing pipelines, lakehouse models, governance, and analytics-ready assets.
Data Engineer
Data Engineer building Snowflake and Azure claims data infrastructure for ProgenyHealth. Supporting healthcare analytics, reporting, security, and reliable enterprise data pipelines.
Data Engineer
Data Engineer building scalable pipelines and analytics warehouses for Troveo’s licensed AI training data platform. Supporting data modeling, quality, reporting, and reliable delivery to AI teams.
Data Engineering Lead
Data Engineering Lead guiding engineers and consolidating Oxford Global Resources’ Azure and Microsoft Fabric data platform. Owning architecture, ETL reliability, governance, and delivery.
Principal Engineer – Data Retrieval
Principal Engineer architecting web-scale crawling, ingestion, and retrieval infrastructure for NVIDIA’s foundation AI models. Leading distributed systems design, production scaling, and high-quality pretraining data delivery.
IT Data Science Principal, Data Warehouse
Leading enterprise data warehouse, analytics, and AI capabilities for Cornerstone Building Brands, a North American building-products manufacturer. Driving roadmaps, governance, modernization, and multidisciplinary delivery.
Data Engineer
Data Engineer building reliable pipelines for Branch’s fintech platform, which accelerates payments and provides financial services. Using Python, SQL, Airflow, Snowflake, DBT, and GCP.
Data Engineer
Data Engineer building reliable pipelines and transformations for Definitive Healthcare’s healthcare data products. Supporting Claims and Consumer domains with Python, SQL, Spark, and cloud platforms.
Senior Data Platform Engineer
Senior Data Platform Engineer building scalable data lakes, ingestion pipelines, and Kubernetes infrastructure. Supporting analytics, product applications, AI workloads, and large-scale model execution.
Medical Receptionist, Data Migration Specialist
Remote medical receptionist and data migration specialist transferring patient records from Elation Health to Dakota Health. Supporting U.S. practice scheduling, patient calls, and administrative workflows.
Data Engineer
Data Engineer building Azure Databricks pipelines for credit-card data at CGI, a global IT and business consulting firm. Modernizing financial data platforms and controls.
Working Student – Data Engineering
Working Student building Python and SQL data pipelines for AgentixLake’s cloud data and AI engineering projects. Supporting data quality, lakehouse platforms, testing, and monitoring.
Data Engineer – Temporary
Data Engineer building reliable Zuora-to-GCP ETL pipelines for Applaudo’s AI-powered solutions. Maintaining integrations, data quality, monitoring, and warehouse validation in an async distributed team.
WEKA Data Platform Administrator
WEKA Data Platform Administrator operating NeuralMesh enterprise AI data infrastructure for GPU-cloud customers. Managing storage, Kubernetes, networking, automation, and customer workload performance.
Analytics Engineer – Data Platform
Analytics Engineer building Snowflake and dbt models for Aline’s senior-living resident lifecycle platform. Creating governed data foundations for BI reporting and AI-driven care outcomes.
Senior Data Engineer – Mid Shift
Data Engineer building AWS and Snowflake platforms for Shore360, an Australian-owned Philippine staffing and administrative support firm. Developing reliable pipelines, APIs, and data quality controls.
Senior Software Engineer, Data Infrastructure – RDBMS
Senior engineer operating petabyte-scale PostgreSQL infrastructure for TRM Labs’ AI-powered crime-intelligence platforms. Automating database operations, optimizing performance, and managing CDC pipelines remotely across LATAM.