Senior Director, Data Engineering

Job Description:

The Senior Director Data Engineer will serve as a core architect and enabler of our evolving data ecosystem. You will help design and scale the foundational infrastructure for our global clients, transforming fragmented, reactive data processes into proactive, intelligent systems powered by AI and a robust semantic layer. You will work closely with Analytics, Media, Product, and Engineering teams to optimize data pipelines, automate complex transformations, embed trust and visibility into every stage of the data lifecycle, and enable AI-driven insights that address "why" questions through enhanced reasoning and accuracy.   

Responsibilities 

• Build, scale, and maintain robust data pipelines/models using DBT, Python, PySpark, Databricks, and SQL, integrating AI-first foundations and semantic layers for consistent data interpretation.  

• Design and manage semantic models, star schemas, ontologies, taxonomies, knowledge graphs, and glossaries using DBT YAML, GitHub, Unity Catalog, Fabric/OneLake, and Power BI for unified understanding and AI reasoning.  

• Utilize low-code/no-code tools (Trifacta, DBT, Power BI, Tableau, Fabric/OneLake, Copilot Studio) to build governed semantic layers supporting natural language querying, vector search, and hybrid AI indexing.  

• Own AI deployment pipelines with containerized agents and automation using Kubernetes, n8n, LangChain, Azure AI Foundry, and Copilot Platform (MCP) for multi-step retrieval, summarization, and notifications.  

• Strengthen AI accuracy/governance via metadata, access controls, and grounding (vector DBs, search indexes, knowledge graphs) to deliver reliable responses, source citation, and “why” reasoning.  

• Design modular, reusable data models for analytics, reporting, AI enablement, and agentic apps, including LLM integration for intent parsing, routing, retrieval, and synthesis.  

• Develop and monitor mapping tables, validation rules, lineage, error logging, and observability for ETL/ELT health, data integrity, schema control, and real-time quality monitoring.  

• Collaborate with analysts, engineers, and stakeholders to transform raw data into governed datasets, leveraging Adverity for multi-source integration and normalization.  

• Implement agentic AI and Copilot integrations to enhance data accessibility, autonomous resolution, and dynamic insights across processes.  

• Drive innovation in Data Quality Suite roadmap, including real-time monitoring, dynamic interfaces, self-serve tools, and AI-enhanced features for scalability.  

• Contribute to medallion architecture (bronze/silver/gold), best practices for reusable components, semantic layer extension (e.g., RAG indexing), and AI infrastructure.  

• Manage Databricks Unity Catalog, Workflows, SQL Analytics, Notebooks, and Jobs for governed analytics and ML workflows.  

• Develop pipelines/tools with Microsoft Fabric, Power BI, Power Apps, Azure Data Lake/Blob, and Copilot Studio, tied to GitHub, n8n, and Kubernetes orchestration.  

• Leverage GitHub and GitHub Copilot for version control, CI/CD, automation, code suggestions, and collaboration on SQL, Python, YAML, and agent development.  

• Utilize Java or Scala for custom processing scripts, scalable ingestion, and advanced AI actions like code execution and vector search. 

Qualifications 

• 8+ years of experience as a Data Engineer or in a similar role building scalable data infrastructure, with at least 2+ years focused on AI-integrated systems, semantic layers, or agentic AI deployments. 

• Bachelor's Degree in Computer Science, Engineering, Information Systems, or related field required.

• Advanced expertise in SQL, Python, DBT; strong experience with PySpark, Databricks, and semantic layer tools like DBT YAML, Unity Catalog, and knowledge graphs required. 

• Hands-on experience with ETL/ELT design tools like Trifacta (Alteryx), Adv