Databricks Architect (Lead Data Platform Architect)

Role Overview

We are looking for a highly skilled Databricks Architect to design, build, and scale enterprise-grade Lakehouse data platforms . This role will drive architecture strategy, platform standardization, and enterprise data modernization initiatives , leveraging Databricks and cloud ecosystems.

The ideal candidate brings deep expertise in Spark, Delta Lake, and cloud-native architecture , along with strong leadership in driving large-scale data transformations.

Key Responsibilities

Data Platform Architecture

• Define and implement end-to-end Databricks Lakehouse architecture .
• Design scalable systems for:
  • Batch & real-time data processing
  • Structured & unstructured workloads
• Establish medallion architecture (Bronze, Silver, Gold layers) as a standard.

Databricks Platform Leadership

• Lead deployment and optimization of:
  • Azure Databricks / AWS Databricks / GCP Databricks
• Define standards for:
  • Workspace design & cluster strategy
  • Job orchestration
  • Data storage (Delta Lake)
• Drive adoption of:
  • Unity Catalog
  • MLflow
  • Databricks SQL & Photon

Solution Design & Engineering

• Architect robust data ingestion frameworks :
  • Batch (ADF, Airflow)
  • Streaming (Kafka, Event Hub)
• Define reusable patterns for:
  • ETL/ELT pipelines
  • Data modeling (star schema, data vault, dimensional models)
• Guide engineering teams on best practices in Spark/PySpark optimization .

Performance & Cost Optimization

• Optimize workloads for:
  • Query performance
  • Cluster utilization
  • Storage efficiency
• Implement cost governance strategies (auto-scaling, job clusters, spot instances).

Data Governance & Security

• Architect enterprise-grade governance frameworks:
  • Data lineage, cataloging, metadata management
  • Fine-grained access control (RBAC/ABAC)
• Ensure compliance with data privacy and regulatory standards .

Cloud & Ecosystem Integration

• Integrate Databricks with:
  • Data sources (ERP, CRM, APIs, IoT)
  • BI tools (Power BI, Tableau)
  • ML pipelines and AI platforms
• Collaborate with cloud architects for:
  • Networking, security, and storage strategies.

Leadership & Mentorship

• Provide architectural guidance to data engineers, scientists, and TPMs .
• Conduct design reviews and enforce architecture governance .
• Mentor teams on emerging patterns:
  • Data Mesh
  • DataOps / MLOps
  • GenAI workloads on Databricks

Skills & Qualifications

Mandatory Skills

• 12+ years of experience in data engineering, architecture, or platform design .
• 5+ years of hands-on experience with:
  • Databricks (must-have)
  • Apache Spark / PySpark / SQL
• Strong expertise in:
  • Delta Lake
  • Distributed data processing
• Experience with at least one cloud:
  • Azure (preferred), AWS, or GCP