Part-Time Senior Data Platform Engineer – Azure Databricks & API Integration

FlexStaff is seeking a Senior Data Platform Engineer - Azure Databricks & API Integration (PHI)

• This is a temporary role, for a one -year assignment
• Part-Time - approximately 20 hours/week
• Only candidates who reside in New York State can be considered.

The Senior Data Platform Engineer will lead the operation, stability, and evolution of our client's Azure Databricks environment and the API integrations that support enterprise data exchange involving Protected Health Information (PHI). This role is a key senior individual contributor within the Data Engineering organization and works closely with architecture, security, and application teams to ensure a secure, scalable, and compliant data platform.

The ideal candidate brings deep hands-on experience with Azure Databricks, Azure-native services, and API development, along with a strong understanding of healthcare data security and compliance.

Key Responsibilities

Azure Databricks & Data Platform Ownership

•Own and maintain the Azure Databricks workspace(s), including clusters, jobs, pools, and libraries

•Design, build, and optimize Spark-based data pipelines using Databricks, Delta Lake, and PySpark/Scala

•Implement best practices for cluster sizing, autoscaling, job orchestration, and cost optimization

•Manage Databricks access controls using Azure AD, RBAC, and workspace permissions

•Support CI/CD pipelines for Databricks assets using Azure DevOps and Git integration

API Integration & Data Access

•Design, develop, and support secure RESTful APIs that enable data ingestion and consumption

•Integrate Azure Databricks with upstream and downstream systems using APIs, messaging, or batch interfaces

•Ensure API reliability through proper versioning, monitoring, logging, and alerting

•Collaborate with application and integration teams to meet data access and performance requirements

Security, Compliance & PHI

•Ensure all Databricks workloads and APIs meet HIPAA, Azure security, and enterprise compliance standards

•Implement encryption at rest and in transit, key management (Azure Key Vault), and secrets handling

•Enforce data protection controls including data masking, row/column-level security, and auditing

•Partner with security, privacy, and compliance teams on audits, risk assessments, and remediation

Azure & Platform Operations

•Leverage Azure services such as ADLS Gen2, Azure Data Factory, Event Hubs, Azure Monitor, and Log Analytics

•Monitor platform health, performance, and availability; troubleshoot and resolve production issues

•Create and maintain technical documentation, standards, and operational runbooks

•Provide technical guidance and mentorship within the Data Engineering team

 

Required Skills & Qualifications

Technical Skills

•Strong hands-on experience with Azure Databricks (administration and development)

•Advanced proficiency in Python (PySpark) and/or Scala

•Experience developing and supporting REST APIs (e.g., Python/FastAPI, .NET, or Java/Spring Boot)

•Deep experience with Azure data and integration services, including:

o        Azure Data Lake Storage (ADLS Gen2)

o        Azure Data Factory or Synapse pipelines

o        Azure Event Hubs or Service Bus

 

•Experience with Azure DevOps, Git, and CI/CD automation

•Infrastructure as Code experience using Terraform or ARM/Bicep

•Strong understanding of cloud security, identity, and access management

Compliance & Domain Knowledge

•Proven experience working with PHI and regulated healthcare data

•Strong knowledge of HIPAA and data privacy best practices

•Experience implementing audit logging, access controls, and governance in Azure

Experience & Background

•7+ years of experience in data engineering, platform engineering, or backend engineering

•3–5+ years of hands-on Azure Databricks experience in production environments

•3+ years of experience designing and supporting APIs at enterprise scale

•Experience in healthcare, life sciences, or other highly regulated industries strongly preferred

•Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience

Preferred Qualifications

•Experience with Delta Live Tables, Unity Catalog, or Databricks Workflows

•Familiarity with FHIR, HL7, or healthcare interoperability standards

•Experience with streaming architectures and near–real-time data processing

•Azure certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect)