Senior Data Engineer (Databricks)

<p><strong>We are looking for a skilled and proactive Senior Data Engineer with deep expertise in Databricks to join our data platform team. You will be responsible for designing, building, and optimizing scalable data pipelines and lakehouse architectures that power analytics, reporting, and machine learning across the organization</strong><br/></p> <p><strong>Responsibilities</strong></p> <ul> <li>Develop and maintain robust ETL/ELT pipelines using Databricks and Apache Spark</li> <li>Design and implement Delta Lake architectures for structured and semi-structured data</li> <li>Collaborate with data analysts, scientists, and product teams to deliver clean, reliable datasets</li> <li>Optimize performance of Spark jobs and manage cluster resources efficiently</li> <li>Automate workflows using Databricks Jobs, Workflows</li> <li>Ensure data quality, lineage, and governance using Unity Catalog and monitoring tools</li> <li>Document data models, pipeline logic, and architectural decisions</li> <li>Participate in code reviews and contribute to engineering best practices</li> </ul> <p><strong> Required skills and experience</strong></p> <ul> <li>4+ years of experience as a Data Engineer or in a similar role</li> <li>Strong hands-on experience with <strong>Databricks</strong>, including Delta Lake, Spark SQL</li> <li>Proficiency in <strong>Python</strong> and <strong>SQL</strong> for data manipulation and pipeline development</li> <li>Solid understanding of <strong>Apache Spark</strong> internals and performance tuning</li> <li>Experience with <strong>cloud platforms</strong> (Azure, AWS,GCP)</li> <li>Knowledge of data modeling, partitioning, and lakehouse principles</li> <li>Ability to work with large-scale datasets and optimize storage and compute costs</li> <li>Strong communication skills and ability to collaborate across teams</li> </ul> <p><strong>Nice to have</strong></p> <ul> <li>Experience with <strong>Azure Data Factory</strong> or <strong>Airflow</strong></li> <li>Proficiency with modern data warehouses and tools including <strong>Snowflake</strong>, <strong>Synapse, Red Gate </strong></li> <li>Exposure to <strong>data governance</strong> frameworks and tools (e.g., Unity Catalog, Purview)</li> <li>Understanding of <strong>machine learning workflows</strong> and integration with data pipelines</li> <li>Experience with <strong>BI tools</strong> (Power BI, Tableau) and supporting analytics teams</li> <li>Contributions to open-source projects or technical blogs</li> </ul> <p><br/></p>