Job Description:

Job Description: Data Engineer – Databricks & Microsoft Fabric

Location : Chennai, India
Experience : 6 to 8 years

Role Summary
We are looking for an experienced Data Engineer with strong hands-on expertise in Databricks, Microsoft Fabric, PySpark, SQL, and cloud-based data engineering. The candidate will be responsible for designing, developing, and optimizing scalable data pipelines, lakehouse solutions, and analytics-ready data models.
The role requires strong experience in building end-to-end data engineering solutions, working with structured and semi-structured data, implementing data quality controls, and supporting enterprise reporting and analytics platforms.
Key Responsibilities
Data Engineering and Pipeline Development
•    Design, build, and maintain scalable data pipelines using Databricks, PySpark, Spark SQL, and Microsoft Fabric.
•    Develop batch and incremental data ingestion pipelines from multiple source systems.
•    Build and optimize Bronze, Silver, and Gold layer data models using lakehouse architecture.
•    Implement ELT/ETL workflows for data transformation, enrichment, validation, and publishing.
•    Work with structured, semi-structured, and unstructured data formats such as CSV, Parquet, JSON, Delta, and XML.
Databricks Development
•    Develop notebooks, jobs, workflows, and reusable components in Azure Databricks.
•    Implement Delta Lake features such as schema evolution, merge/upsert, time travel, and optimized storage.
•    Optimize Spark jobs for performance, scalability, and cost efficiency.
•    Implement partitioning, caching, indexing, and cluster optimization strategies.
•    Troubleshoot job failures, performance bottlenecks, and data quality issues.
Microsoft Fabric Development
•    Build data solutions using Microsoft Fabric Lakehouse, Warehouse, Data Factory, Pipelines, Notebooks, and OneLake.
•    Develop and manage data pipelines in Fabric for ingestion, transformation, and orchestration.
•    Work with Fabric SQL endpoints, semantic models, and Power BI integration.
•    Support migration or modernization of existing data platforms into Microsoft Fabric.
•    Implement reusable data engineering patterns and framework-based development in Fabric.
Data Quality, Governance, and Security
•    Implement data validation, reconciliation, exception handling, and audit controls.
•    Define and apply data quality rules including null checks, duplicate checks, referential checks, and cross-field validations.
•    Maintain data lineage, metadata, source-to-target mapping, and technical documentation.
•    Ensure data pipelines comply with enterprise security, access control, and governance standards.
•    Support integration with data governance tools such as Microsoft Purview, where applicable.
DevOps and Production Support
•    Implement CI/CD practices for notebooks, pipelines, SQL scripts, and configuration files.
•    Use Git-based version control and deployment processes across environments.
•    Monitor production jobs and resolve incidents within agreed timelines.
•    Prepare runbooks, deployment guides, operational support documents, and handover materials.
•    Collaborate with architects, business analysts, data analysts, and reporting teams to deliver reliable data solutions.
Required Skills
Technical Skills
•    Strong hands-on experience in Azure Databricks.
•    Strong experience in Microsoft Fabric components such as Lakehouse, Warehouse, Data Factory, Pipelines, Notebooks, and OneLake.
•    Proficiency in PySpark, Spark SQL, Python, and SQL.
•    Strong knowledge of Delta Lake, lakehouse architecture, and medallion architecture.
•    Experience with cloud storage and data platforms, preferably Azure Data Lake Storage, Azure SQL, Synapse, or Fabric OneLake.
•    Experience in data ingestion from databases, APIs, files, SFTP, cloud storage, and streaming sources.
•    Good understanding of data modeling, dimensional modeling, and analytics-ready data structures.
•    Experience with performance tuning of Spark jobs and SQL queries.
•    Experience in job scheduling, monitoring, logging, and error handling.
•    Knowledge of CI/CD, Git, Azure DevOps, and deployment automation.
Preferred Skills
•    Experience with Power BI and semantic model integration.
•    Experience in migrating workloads from legacy ETL tools, Synapse, ADF, or Databricks to Microsoft Fabric.
•    Knowledge of Microsoft Purview for data cataloging, lineage, and governance.
•    Experience in building reusable data engineering frameworks.
•    Exposure to real-time or near-rea