Lead Databricks Data Engineer

You could be the one who changes everything for our 28 million members by using technology to improve health outcomes around the world.  As a diversified, national organization, Centene's technology professionals have access to competitive benefits including a fresh perspective on workplace flexibility.
 

Position Purpose:

Uses advanced expertise and knowledge to lead the development and operationalization of data pipelines to make data available for consumption (reports and advanced analytics), including data ingestion, data transformation, data validation / quality, data pipeline optimization, and orchestration. Works closely with the DevSecOps Engineer during continuous integration and continuous deployment.

• Leads the design and implementation of standardized data management procedures around data staging, data ingestion, data preparation, data provisioning, and data destruction (scripts, programs, automation, assisted by automation, etc.)

• Leads data engineers and provides advanced experience in the design, development, implementation, testing, documenting, and operating of large-scale, high-volume, high-performance data structures for business intelligence analytics

• Leads the team using advanced experience and knowledge in the design, development, and maintenance of real-time processing applications and real-time data pipelines

• Ensure quality of technical solutions as data moves across Centene’s environments

• Provides advanced knowledge and insight into the changing data environment, data processing, data storage, and utilization requirements for the company and offers suggestions for solutions

• Develops, constructs, tests, and maintains architectures using advanced programming language and tools

• Leads and implements ways to improve data reliability, efficiency, and quality; use data to discover tasks that can be automated

• Performs other duties as assigned

• Complies with all policies and standards

Education/Experience:

A Bachelor's degree in a quantitative or business field (e.g., statistics, mathematics, engineering, computer science) and requires 5 – 7 years of related experience.
Or equivalent experience acquired through accomplishments of applicable knowledge, duties, scope and skill reflective of the level of this position.

Technical Skills:

• One or more of the following skills are desired.

• Advanced proficiency in implementations using Databricks Lakehouse Medallion architecture and Unity Catalog (for data governance, lineage tracking, and access control).

• Skilled in architecture/design/build of scalable data pipelines using PySpark, SQL and/or Python on AWS Databricks.

• Proficient in handling relational, flat files and hierarchical (JSON, XML) data formats.

• Experienced in integrating cloud storage platforms:  S3, Azure Blob, or Databricks Volumes.

• Skilled in deploying production-grade pipelines with Databricks workflows (leveraging CI/CD – Gitlab, DAB etc.),

• Experienced in data pipeline optimization strategies, development of automated monitoring.

• Familiar with relational and dimensional data modeling, especially healthcare clinical data (HL7/FHIR standards).

• Familiar with Databricks API integrations and Connector tools for BI platforms.

Soft Skills:

• Problem Solving: Demonstrated ability to analyze and resolve complex pipeline and data engineering challenges independently.

• Self-starter: Self-motivated to learn and apply new technologies within an evolving data ecosystem. Demonstrates curiosity and handles ambiguity well.

• Communication: Capable of translating technical concepts into appropriate specs/insights for technical and non-technical stakeholders. Ability to present recommendations to leadership.

• Project Management: Expert at planning, prioritizing, and managing technical deliverables for multiple concurrent projects. Demonstrated track record of accountability.

• Collaboration: Proactive cross-functional collaboration across technical and business teams to align goals and produce successful outcomes.

• Accuracy & Governance: Exceptional attention to detail to ensure precision/quality in data engineering outputs while maintaining compliance with enterprise governance standards.

Pay Range: $100,900.00 - $186,800.00 per year

Centene offers a comprehensive benefits package including: comp