Azure Data Factory Technical Lead

Job Description:

We are seeking a seasoned Azure Data Factory (ADF) Technical Lead with proven track to build scalable data integration solutions using ADF (migration or new implementation). The ideal candidate will have hands-on experience building enterprise or Line of business specific data warehouses/marts. 

Key Responsibilities: 

• Design and build Cloud code pipelines using Azure Data Factory. 
• Develop and optimize ETL/ELT CI/CD workflows using ADF pipelines, Data Flows, Linked Services, Integration Runtimes, and Triggers. 
• Leverage PySpark, Kafka, Kenesis and Python for data transformation, cleansing, and enrichment tasks within Azure Synapse or Databricks environments. 
• Work closely with Architect to build solution.  
• Lead team of junior engineers and developers. 
• Performance tuning, monitoring, and troubleshooting of data pipelines and workflows. 

Required Skills & Experience: 

• Minimum Bachelor’s degree in computer science, Information Systems, or related field. 
• Minimum 6+ years of experience in data warehousing/engineering.  
• At least 2 years in Technical Lead role to lead team of 6 – 8 developers. 
• At least 2+ years in Azure Data Factory architecture and implementation (migration or new implementation). 
• Understanding of ADF components: Pipelines, Datasets, Linked Services, Integration Runtime, Data Flows, and Triggers. 
• 5+ years of experience in building and managing enterprise/data warehouse solutions. 
• 2+ years of experience with Azure tool stack. 
• 2+ years’ experience in Python and PySpark, kafka, kinesis for data processing and scripting. 
• Knowledge of data modeling. 
• Experience in data privacy regulations (e.g., GDPR, HIPAA). 
• Experience in Databricks (on Azure) for large-scale data engineering and transformation workflows, including the use of PySpark, Scala, Delta Lake, and MLflow. Familiarity with Notebook-based collaboration and version-controlled data pipelines. 
• Proficiency in SQL (T-SQL or SparkSQL) for developing complex queries, views, stored procedures, and optimization. Solid experience in Python, especially data manipulation libraries like pandas, numpy, and integration with PySpark. 
• Knowledge (hands-on preferred) of Microsoft Fabric (OneLake, Lakehouse, Notebooks, Pipelines) as an interactive environment for unified data analytics and collaborative workflows across Power BI, Synapse, and Data Engineering workloads. 
• Experience with structured, semi-structured (JSON, Parquet), and unstructured data 
• Azure Schema design and optimization f