Senior Databricks Data Engineer

You could be the one who changes everything for our 28 million members by using technology to improve health outcomes around the world.  As a diversified, national organization, Centene's technology professionals have access to competitive benefits including a fresh perspective on workplace flexibility.
 

Position Purpose:

Develops and operationalizes data pipelines to make data available for consumption (reports and advanced analytics), including data ingestion, data transformation, data validation / quality, data pipeline optimization, and orchestration. Engages with the DevSecOps Engineer during continuous integration and continuous deployment.

• Helps guide the design and implementation of complex data management procedures around data staging, data ingestion, data preparation, data provisioning, and data destruction (scripts, programs, automation, assisted by automation, etc.)

• Provides guidance to data engineers in the design, development, implementation, testing, documenting, and operating of large-scale, high-volume, high-performance data structures for business intelligence analytics

• Designs, develops, and maintains real-time processing applications and real-time data pipelines

• Ensure quality of technical solutions as data moves across Centene’s environments

• Provides senior level knowledge and insight into the changing data environment, data processing, data storage, and utilization requirements for the company and drives the team toward viable solutions

• Develops, constructs, tests, and maintains architectures using advanced programming language and tools

• Drives ways to improve data reliability, efficiency, and quality and deploys a solution; use data to discover tasks that can be automated

• Performs other duties as assigned

• Complies with all policies and standards

Education/Experience:

A Bachelor's degree in a quantitative or business field (e.g., statistics, mathematics, engineering, computer science) and Requires 4 – 6 years of related experience.
Or equivalent experience acquired through accomplishments of applicable knowledge, duties, scope and skill reflective of the level of this position.

Technical Skills:

• One or more of the following skills are desired.

• Experienced in implementations using Databricks Lakehouse Medallion architecture and Unity Catalog (for data governance, lineage tracking, and access control).

• Skilled in building and optimizing complex data pipelines using PySpark,  SQL and/or Python on Databricks (or other comparable modern cloud DBMS preferably on AWS).

• Proficient in handling relational, flat files and hierarchical (JSON, XML) data formats on cloud platforms (AWS, Azure, GCP).

• Skilled in deploying production-grade pipelines with Databricks workflows (leveraging CI/CD – Gitlab, DAB etc.),

• Experienced in data pipeline optimization strategies, development of automated monitoring.

• Familiarity with Databricks API integrations and Connector tools for BI platforms.

Soft Skills:

• Problem Solving: Demonstrated ability to analyze and resolve intermediate to complex data engineering challenges using a data driven approach; seeks guidance as needed.

• Self-starter: Self-motivated to learn and apply new technologies. Demonstrates curiosity and handles ambiguity well.

• Communication: Capable of translating technical concepts into appropriate specs/insights for technical and non-technical stakeholders..

• Project Management: Demonstrated track record of accountability in producing own deliverables. Expected to show resilience even under pressure

• Collaboration: Cross-functional collaboration across technical and business teams to  produce successful outcomes.

• Accuracy & Governance: Attention to detail in ensuring high quality results.

Pay Range: $85,300.00 - $158,100.00 per year

Centene offers a comprehensive benefits package including: competitive pay, health insurance, 401K and stock purchase plans, tuition reimbursement, paid time off plus holidays, and a flexible approach to work with remote, hybrid, field or office work schedules.  Actual pay will be adjusted based on an individual's skills, experience, education, and other job-related factors permitted by law, including full-time or part-time status.  Total compensation may also include additional forms of incentives. Benefits may be subject to program eligibility.<