Sr. Data Engineer

This role is responsible for the design, development, and maintenance of data integration, analytics, and reporting solutions that support our animal genetics and bioinformatics workloads. The ideal candidate will possess expertise in Databricks and modern data engineering tools such as Azure Data Factory, combined with hands on experience working with biological, genomic, or other omics datasets. This position requires a proactive, self-motivated, and results-oriented individual with a passion for data, a strong understanding of data architecture and warehousing principles, and an appreciation for bioinformatics workflows in a commercial genetics environment. 

Responsibilities 

Data Integration 

• Design, develop, and maintain robust and efficient ETL/ELT pipelines and processes on Databricks for both operational and bioinformatics datasets (e.g., genomic markers, phenotypic data, laboratory outputs). 

• Ingest, transform, and harmonize structured and semi-structured biological data from lab systems, LIMS, sequencing platforms, and external partners into the enterprise data platform. 

• Troubleshoot and resolve Databricks pipeline errors and performance issues. 

• Optimize data flow performance and minimize data latency across scientific and business use cases. 

• Implement data quality checks, validations, and reconciliation processes within ETL workflows, including domain-specific checks for genomic and phenotypic data. 

Databricks Development 

• Develop and maintain Databricks pipelines, notebooks, and datasets using Python, Spark, and SQL. 

• Optimize Databricks jobs for performance and cost-effectiveness, including largescale bioinformatics and analytics workloads. 

• Integrate Databricks with other data sources and systems, including lab instruments, genomic databases, and on-prem or cloud data stores. 

• Participate in the design and implementation of data lake architectures that support both traditional analytics and bioinformatics pipelines. 

Data Warehousing 

• Participate in the design and implementation of data warehousing solutions to support reporting, analytics, and scientific modeling. 

• Model and curate subject areas for genetics, reproduction, and bioinformatics (e.g., animals, pedigr