Lead Data Engineer- DataBricks

Job Title: Lead Data Engineer – Databricks

Job Summary

We are seeking a Lead Data Engineer with deep expertise in Databricks to architect, build, and lead scalable data engineering solutions on cloud-based lakehouse platforms. The role combines hands-on technical leadership with solution design, mentoring, and close collaboration with architects, BI, and AI teams.

Key Responsibilities

Technical Leadership & Architecture

• Lead the design and implementation of Databricks Lakehouse architectures

• Define medallion architecture (Bronze, Silver, Gold layers) using Delta Lake

• Drive architectural decisions for batch and streaming data pipelines

• Establish coding standards, best practices, and reusable frameworks

Data Engineering & Databricks

• Design and build scalable ETL/ELT pipelines using Databricks (PySpark/SQL/Scala)

• Optimize Spark jobs for performance, reliability, and cost

• Implement Delta Lake features (ACID, time travel, schema enforcement)

• Develop and manage Databricks workflows, jobs, and clusters

Cloud & Platform Integration

• Architect Databricks solutions on Azure (preferred) or AWS

• Integrate Databricks with cloud storage and data services

  • Azure: ADLS, ADF, Synapse

  • AWS: S3, Glue, Redshift

• Enable BI and analytics consumption (Power BI, Tableau)

Governance, Security & DevOps

• Implement data governance using Unity Catalog

• Define RBAC, data access controls, and security best practices

• Enable CI/CD for Databricks using GitHub / Azure DevOps

• Use Infrastructure-as-Code (Terraform) for environment management

Leadership & Collaboration

• Lead, mentor, and grow data engineering teams

• Conduct design and code reviews

• Collaborate with Data Architects, Product Owners, and stakeholders

• Support production releases, monitoring, and incident resolution

Required Skills

Databricks & Big Data

• Expert-level Databricks experience (Azure or AWS)

• Strong Spark / PySpark / Spark SQL expertise

• Delta Lake and Lakehouse architecture

• Streaming (Structured Streaming) experience

Cloud & Data Platforms

• Strong experience with Azure or AWS cloud platforms

• Data orchestration tools (ADF, Airflow, or similar)

• Strong SQL and data modeling skills

DevOps & Automation

• Git-based version control

• CI/CD pipelines for data engineering workloads

• Terraform or similar IaC tools

Preferred Qualifications

• Experience with MLflow and MLOps workflows

• Exposure to Microsoft Fabric or Snowflake

• Databricks certifications (Professional Data Engineer / Architect)

• Experience working in Agile environments

Education

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field

Quick Fit Indicators