Senior Databricks Engineer

Description

DSG is seeking a Senior Databricks Engineer to support a project team in a data modernization effort. The contractor will leverage expertise in PySpark/Spark SQL to refactor legacy SAS/SQL code while developing and configuring ETL and DevOps pipelines for Azure Databricks environments.

This role is on-site in Irving, TX, four days per week, and requires candidates to be legally authorized to work in the U.S. without employer sponsorship.

Key Responsibilities

Refactor Legacy Code – Interpret and refactor legacy SAS/SQL scripts with complex joins and data transformations across diverse data sources.

Data Ingestion & Transformation – Develop PySpark frameworks for ingesting data from multiple sources (CSV, Excel, on-prem servers) using Lakehouse federation.

Azure Infrastructure Development – Create and manage storage containers, compute resources, and Azure pipelines for Databricks environments.

Security & Compliance – Guide infrastructure teams in configuring Unity Catalog/Schemas, workspaces, clusters, and storage containers to meet project needs.

CI/CD Implementation – Develop CI/CD pipelines using Databricks asset bundles and Azure DevOps for streamlined deployment and automation.

Requirements

Required:

? 2-5 years of experience developing ETL pipelines using Databricks, preferably in Azure Databricks.

? Strong programming skills in PySpark, Python, and SQL.

? Deep understanding of data structures and transformations.

? Strong grasp of Azure Databricks architecture.

Preferred:

? Databricks certifications, such as Databricks Engineering Professional or Apache Spark certifications.

Additional Details

Location Requirement: Must be based in the United States.

No employer sponsorship available.