Senior Fabric Engineer

 

About the team: 

Capco’s Data Team helps our clients transform every aspect of their business.  We are highly skilled at formulating data strategy, defining business and technology initiatives across the data management lifecycle, and aligning multi-year strategic roadmaps with client’s business goals. As digital technologies advance and regulations tighten, today’s consumers – and, therefore, today’s businesses – are becoming more aware of the importance of good quality data. We work to establish holistic ways to effectively manage data through the modern data supply chain and facilitate consumption through analytics, modelling, AI, machine learning, dashboarding, and reporting.  

 

About the Job: 

The Data Engineer will serve as the lead technical specialist for designing and implementing data science and advanced analytics capabilities on Microsoft Azure Fabric and Databricks. This role focuses on data processing, identity resolution, entity linking, and data warehouse development that enable organizations to unify fragmented data across multiple systems into a trusted, governed, and analytics-ready model. The ideal candidate combines deep hands-on expertise in Databricks engineering, data modeling, and applied data science, with the ability to build scalable, production-grade data solutions in collaboration with business, engineering, and analytics teams. 

 

What You’ll Get to Do: 

Data Platform & Warehouse Development 

• Design and develop data lakehouse and warehouse structures within Azure Databricks and Fabric environments. 

• Build ETL and ELT pipelines to extract, cleanse, normalize, and enrich data from CRM, ERP, LMS, and financial systems . 

• Develop reusable data transformation and validation frameworks leveraging PySpark, SQL, and Delta Live Tables. 

• Support the operationalization of the central data warehouse using Azure SQL and Fabric Data Warehouse. 

Identity Resolution & Data Linking 

• Implement entity resolution models to unify customer, member, or participant records across systems using deterministic and probabilistic matching techniques. 

• Design and deploy matching algorithms utilizing Databricks MLflow, PySpark, and Azure Machine Learning for cross-system deduplication and linkage. 

• Collaborate with architects to define unique identifiers, external keys, and golden record frameworks for enterprise data integration. 

• Monitor and continuously refine data matching accuracy, precision, and recall metrics. 

Data Processing & Automation 

• Develop and schedule data ingestion pipelines in Azure Fabric and Databricks for recurring Excel, CSV, and structured PDF sources using Power Automate, Form Recognizer, and Fabric Dataflows. 

• Apply data quality and validation rules to flag incomplete, inconsistent, or stale records. 

• Build and automate data lineage, change tracking (CDC), and error-handling workflows. 

• Support performance tuning and scalability for high-volume processing environments. 

Analytics & Modeling Support 

• Provide curated and feature-engineered datasets for Power BI dashboards and machine learning use cases. 

• Partner with data analysts to define KPIs and enable cross-system reporting and predictive insights. 

• Develop scripts and notebooks to support exploratory data analysis (EDA) and visualization in Databricks. 

 

What You’ll Bring with You: 

• BA in Data Science, Computer Science, Applied Mathematics, or related discipline. 

• 5+ years of experience in data engineering and applied data science on Azure platforms. 

• 3+ years building and managing pipelines in Azure Databricks (PySpark, Delta Lake, MLflow). 

• 2+