Databricks Migration Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Databricks Migration Engineer migrating government data systems from SQL Server and ADF to Databricks Lakehouse. Designing secure, optimized Delta Lake pipelines and Power BI data models.
Responsibilities:
- Lead the technical migration from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake)
- Translate relational data warehousing paradigms into scalable, distributed Lakehouse frameworks across Bronze, Silver, and Gold layers
- Design reusable ETL/ELT frameworks using PySpark, Delta Live Tables, and Databricks Workflows
- Architect and refine Gold Layer dimensional models and star schemas for Power BI performance
- Optimize Databricks SQL Warehouses for high-concurrency, low-latency Power BI queries
- Implement Z-Ordering, data skipping, liquid clustering, and materialized views
- Define governance standards for cluster sizing, auto-scaling, and serverless SQL compute
- Monitor Databricks Unit consumption and identify cost-saving opportunities
- Establish Delta Lake partitioning and file size management practices
- Design and implement Unity Catalog data security models
- Enforce row-level and column-level security policies
- Align Lakehouse security with Azure Active Directory/Microsoft Entra ID and RBAC standards
- Serve as primary technical lead through pair-programming sessions, workshops, and code reviews
- Create technical documentation, architecture diagrams, design patterns, and optimization playbooks
- Build knowledge transfer processes for post-migration team self-sufficiency
- Communicate with technical and non-technical audiences
- Complete tasks in an organized, timely, and detail-oriented manner
Requirements:
- Bachelor's degree or higher from an accredited college or university in Computer Science, Engineering, or a related technical field
- 5+ years of experience in Data Engineering, Data System Development or related roles
- 5+ years of experience with Cloud platforms (e.g. Azure, AWS, GCP)
- 1+ year leading complex, cross-functional data projects and technical teams
- Experience with Databricks Lakehouse, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms
- Experience with data warehousing, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark
- Excellent communication skills
- Must be a US Citizen and be able to obtain a Position of Public Trust Clearance
- Must have resided in the US for the last 5 years and not have traveled outside the US for a combined total of 6 months or more in last 5 years
Benefits:
- Contract-to-Hire position
- Remote work arrangement
- East Coast time zone candidates preferred
Similar Jobs
Staff Forward Deployed Engineer
Staff Forward Deployed Engineer building AI-powered client integrations for OpenLoop’s telehealth infrastructure. Creating reusable SDKs, agents, API tooling, and deployment practices across U.S. virtual care.
Specification Engineer – HVAC
Specification Engineer expanding HVAC cooling and heating solutions across Spain’s Levante region. Advising customers on efficient climate-control products while driving product specification growth.
Senior Technical Services Engineer – Remote US West
Senior Technical Services Engineer troubleshooting WEKA enterprise infrastructure for top-tier customers. Bridging customer needs with Engineering while supporting storage, networking, virtualization, and cloud environments.
Senior Technical Services Engineer – Remote US West
Senior Technical Services Engineer supporting WEKA’s enterprise infrastructure customers. Troubleshooting Linux, networking, storage, cloud, and HPC environments while bridging customers and Engineering.
Designated Services Engineer
Senior services engineer supporting enterprise customers using WEKA’s AI-native data infrastructure. Troubleshooting distributed storage, networking, and cloud systems while bridging customers and Engineering.
Exposure Management Engineer
Exposure Management Engineer deploying Axonius-based asset and vulnerability management for GuidePoint Security, a cybersecurity solutions provider. Automating remediation workflows and improving clients' cyber risk visibility.
Technical Success Engineer
Technical Success Engineer helping investment firms maximize Addepar’s financial data and AI platform. Delivering client solutions, automating workflows, and partnering cross-functionally to improve investment operations.
Lead Program Engineer – Major Projects Group
Lead program engineer integrating renewable, gas, storage, and data center infrastructure for Blattner. Coordinating engineering scopes, design reviews, budgets, procurement, and construction support.
Occupational Health and Safety Engineer
Engenheiro de segurança do trabalho remoto na Capco, consultoria empresarial. Atuando com auditorias, análises de riscos e simulados de emergência em plantas industriais.
Senior Forward Deployed Engineer
Senior Forward Deployed Engineer building customer workflows for PackageX’s Vision AI-powered physical logistics platform. Owning enterprise discovery, solution configuration, pilots, and commercial close.
Senior Scientist / Engineer – Synthetic Biology, Bioengineering
Senior scientist/engineer advancing synthetic biology and bioengineering R&D for government and life-science clients. Evaluating technologies, guiding programs, and translating research toward real-world impact.
Geotechnical Project Engineer
Remote geotechnical engineer managing large infrastructure projects for Terracon, a nationwide engineering consulting firm. Delivering site analysis, designs, reports, budgets, and construction support.
Field PLC Engineer
Field PLC Engineer developing and commissioning PLC automation for Divcon’s HVAC, power, and mission-critical data center systems. Client-facing role requiring frequent travel to U.S. job sites.
Forward Deployed Engineer – Immuware
Forward Deployed Engineer building configured and coded Immuware solutions for Propelus healthcare customers. Translating healthcare compliance needs into scalable workflows and integrations.
Senior AdTech Engineer
Senior AdTech Engineer building low-latency programmatic advertising systems and AI media-buying agents. Smart Working connects skilled professionals with global teams through full-time remote roles.
Senior Technical Services Engineer
Senior Technical Services Engineer supporting HealthEdge’s healthcare ETL pipelines and software applications. Troubleshooting production issues, optimizing databases, and coordinating deployments across teams.
Associate Technical Services Engineer
Associate Technical Services Engineer supporting HealthEdge’s healthcare insurance technology customers. Troubleshooting Java, SQL, database, and web-service issues while resolving support tickets within SLAs.
Senior Identity Engineer
Senior Identity Engineer owning SailPoint workstreams, integrations, and identity governance delivery. Building enterprise IAM solutions for Palyrian, a U.S.-based identity services boutique.
Manufacturing Development Engineer
Manufacturing Development Engineer advancing NVIDIA hardware assembly readiness, DFA, fixtures, and automation. Partnering with CEMs from NPI through mass production.
Level 3 Engineer, Microsoft 365 Specialist
Level 3 Engineer strengthening Microsoft 365, RMM, PSA, and security systems for a South Australian managed services provider. Delivering advanced technical support across small-business client environments.