DE&A - Core - Cloud Data Engineering - Azure Data Engineering

Role Summary

We are looking for a strong Data Engineering Architect with 12–16 years of experience in building and architecting modern data platforms on Microsoft Azure. The ideal candidate will have deep hands-on expertise in Azure Data Factory (ADF) pipeline engineering, SQL performance tuning, and end-to-end data integration architecture, along with a strong analytical mindset to troubleshoot complex data issues. You will lead solution architecture, define best practices, and mentor teams to build scalable, secure, and reliable data solutions.

Key Responsibilities

Azure Data Engineering & Architecture

• Architect and design end-to-end Azure data engineering solutions (batch + near real-time) aligned to enterprise standards.
• Define target state architecture for data ingestion, transformation, orchestration, and serving layers.
• Lead architectural decisions around scalability, resiliency, performance, security, governance, and cost optimization.

Azure Data Factory (ADF) – Pipeline Engineering (Core)

• Design, develop, test, and deploy Azure Data Factory pipelines following best practices (modular design, parameterization, reusability, CI/CD readiness).
• Build robust ingestion and orchestration workflows using:
  • Linked Services, Datasets, Pipelines, Triggers
  • Mapping Data Flows / Wrangling Data Flows (where applicable)
  • Integration Runtime strategies (Self-hosted / Azure IR)
• Implement operational excellence: logging, alerting, retry patterns, failure handling, and idempotent design.

SQL Development & Optimization

• Develop and optimize SQL queries and stored procedures to support ADF pipeline operations and downstream transformations.
• Conduct query plan analysis and performance tuning (indexes, partitioning strategies, statistics, query rewrites).
• Establish SQL coding standards and reusable patterns for transformation logic.

Troubleshooting & Analytical Problem Solving

• Apply a strong analytical mindset to diagnose and resolve complex data integration issues across ingestion, transformation, orchestration, and storage layers.
• Perform root cause analysis (RCA) for pipeline failures, performance degradation, data quality issues, and environment instability.
• Design proactive monitoring dashboards and alerts for pipeline SLAs and data freshness.

Engineering Standards, DevOps & Governance

• Define and enforce best practices for:
  • CI/CD for ADF (Azure DevOps / Git-based workflows)
  • Infrastructure-as-Code (ARM/Bicep/Terraform—preferred)
  • Version control, code review, release management
• Implement data governance patterns: metadata management, lineage, auditing, encryption, RBAC, key management, PII controls.
• Collaborate with security/compliance teams to ensure enterprise adherence.

Leadership & Stakeholder Management

• Act as a technical leader for data engineering squads; mentor and guide engineers on design patterns and implementation.
• Translate business requirements into technical architecture and delivery plans.
• Work closely with Product Owners, Data Analysts, Data Scientists, and Platform teams to ensure alignment.

Required Skills & Qualifications

Must-Have (Strong)

• 12–16 years of overall IT experience with significant data engineering & architecture exposure.
• Strong Azure Cloud Data Engineering and associated services architecture knowledge
• Deep hands-on experience with:
  • Azure Data Factory (ADF) – pipeline design, orchestration, integration runtime strategy
  • SQL – advanced querying, stored procedures, performance tuning
• Strong troubleshooting skills for complex multi-system data issues.
• Strong understanding of data architecture concepts:
  • Data lakes/lakehouse/warehouse, dimensional modeling, ELT/ETL patterns
  • Batch orchestration, dependency management, SCD handling, incremental loads

Azure Ecosystem (Preferred / Good to Have)

• Azure data services experience in one or more:
  • Azure Synapse Analytics / Dedicated SQL Pools
  • Azure Databricks / Spark
  • ADLS Gen2, Azure SQL DB, Managed Instance
  • Event Hub / Kafka, Stream Analytics (if real-time involved)
• Monitoring & observability:
  • Azure Monitor, Log Analytics, Application Insights
• Security & identity:
  • Azure AD, Managed Identity, Key Vault, RBAC
  <