Senior Engineer, Data Platforms

Arcadia is dedicated to happier, healthier days for all. We transform diverse data into a unified fabric for health. Our platform delivers actionable insights for our customers to advance care and research, drive strategic growth, and achieve financial success. For more information, visit arcadia.io.
Why This Role Is Important to Arcadia
Arcadia’s business is anchored by its scalable data processing infrastructure and pluggable analytics and data enrichment platform. We collect and store health care data at petabyte scale and have built the tools to enable rapid development and deployment of novel analytics capabilities and content, bringing together a legacy of Arcadia data and modern computation engines. To make this vision a reality, Arcadia has simplified the technologies and databases used to integrate with these computation engines and provides a state-of-the-art development ecosystem for its engineers.    In this role, you will help take this platform to the next level by building out cloudevent microservices and Apache Spark applications that more efficiently move, transform, enrich, or analyze Arcadia’s key data assets. You will help Arcadia provide actionable insights to our health care partners and thereby improve the care for millions of patients. 
What Success Looks LikeIn 3 months- Demonstrated understanding of Arcadia’s healthcare data and high-level system architecture- Actively contributing to the development of CloudEvent Microservices and/or Spark applications - Creating and troubleshooting data platform service and application deployment and orchestration- Strong grasp of Arcadia’s SDLC and engineering processes- Positive contributor to technical and architectural discussions- Fundamental understanding of Semantic/Linked Data and FHIR
In 6 months- Deep familiarity with Arcadia’s data enrichment and analytics systems- Comfortable with Arcadia’s unified Kubernetes ecosystem for Spark and CloudEvent Microservices- Proficient in Arcadia’s CI/CD pipelines, including GitHub Actions, ArgoCD, and Helm- Fully proficient in developing CloudEvent Microservices and Spark applications- Collaborating with Architects to design solutions to new development projects
In 12 months - Highly proficient in core components of the Arcadia data platform and approach to deploying and managing production applications- Acting as tech lead on new development projects, including end-to-end ownership of software development and delivery- Identifying areas for technical improvement and innovation 

What You'll Be Doing

• Working alongside a talented engineering team to design and implement horizontally scalable big data systems processing over a billion records nightly
• Utilizing cloud-native technologies like Kubernetes, Kafka, Spark, Hudi, Cassandra (Scylla), CloudEvent Microservices, and Nifi - Advancing Arcadia’s flagship product on the AWS cloud platform
• Leading full-stack design and development of key software components for Arcadia’s platform
• Architecting and building cloud-event microservices and Spark applications for real-time and batch processing
• Collaborating with engineers, QA analysts, product specialists, and subject matter experts to deliver quality solutions
• Debugging and responding to critical production issues impacting customer workflows
• Mentoring junior engineers and contributing to their growth and development
• Reviewing code, contributing to architectural decisions, and champion best practices
• Documenting architectural designs and development tasks to support the engineering lifecycle
• Driving the adoption of emerging development tools to maintain efficient local development environments
• Leveraging CI/CD tools like Gradle, GitHub Actions, and ArgoCD to streamline deployment

What You'll Bring

• Passionate about solving complex, large-scale distributed systems problems and possesses a deep understanding of distributed computing, parallel processing, and big data tech
• 5+ years of experience in distributed systems and cloud-native technologies
• Proficiency with Java, Spark, and Kubernetes
• Extensive experience with cloud platforms (AWS, GCP, or Azure)
• Familiarity with Data Lake and Lakehouse architecture
• Strong knowledge of Agile software development processes and CI/CD pipelines
• Experience mentoring junior engineers and influencing architectural decisions

Would Love For You To Have

• Experience working with Open Table Formats (Hudi, Iceberg, Delta)
• Familiarity with healthcare