Sr. Data Engineer - Lakehouse Architecture

At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our employees around the world work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work, and we put people first. We’re looking for people who are determined to make life better for people around the world.

Tech@Lilly is seeking a highly skilled Senior Data Engineer who can implement and optimize large-scale Lakehouse solutions and drive the evolution of our modern data platform while providing technical leadership to a growing team. The ideal candidate will have hands-on experience with modern data engineering technology stack and a proven track record of managing engineering talent in fast-paced environments.

What you will be doing

• Design and implement comprehensive Lakehouse architecture solutions using technologies like Databricks, Snowflake, or equivalent platforms 
• Build and maintain real-time and batch data processing systems using Apache Spark, Kafka, and similar technologies
• Architect scalable data pipelines that handle structured, semi-structured, and unstructured data to deliver AI ready data.
• Develop data transformation workflows using tools like DBT, Airflow, or Databricks 
• Lead the technical strategy for data lake and data warehouse integration, ensuring optimal performance and cost efficiency.
• Implement data governance frameworks, including data quality monitoring, lineage tracking, data time travel and security protocols.
• Implement centralized data catalog system and enhance data discovery using technologies like Elastic Search / Open Search.
• Establish monitoring and alerting systems for data pipeline health using technologies like Apache Superset 
• Drive adoption of modern data engineering best practices including Infrastructure as Code, CI/CD, and automated testing
• Collaborate with data scientists, analysts, and business stakeholders to translate requirements into robust technical solutions
• Mentor a team of 3-5 data engineers
• Foster a collaborative team culture focused on continuous learning and innovation

How you will succeed

• Proven ability to mentor junior engineers and facilitate knowledge sharing
• Strong project management skills with experience leading cross-functional initiatives
• Coordinate cross-functional projects and ensure effective communication between technical and business teams
• Demonstrated ability to make architectural decisions and drive technical consensus
• Embrace a growth mindset and actively seek opportunities to expand your leadership capabilities and technical mastery

What you should bring

• Experience with streaming data technologies (Kafka)
• Familiarity with data cataloging tools (Apache Atlas or DataHub)
• Familiarity with high performance data service framework (Arrow Flight)
• Industry certifications in cloud platforms or big data technologies
• Expert-level proficiency in Python and SQL for data transformation and pipeline development
• Strong experience with Apache Spark for big data processing and analytics
• Hands-on experience with cloud platforms (AWS or Azure) and their data services
• Proficiency with Infrastructure as Code tools (Terraform, CloudFormation)
• Experience with containerization (Docker, Kubernetes) and orchestration platforms
• Knowledge of data modeling techniques for both analytical and operational workloads
• Understanding of data governance, security, and compliance requirements
• Knowledge in the pharmaceutical or life sciences domain

Basic Qualification

• Master’s degree in computer science, Engineering, or related technical field
• 3+ years of hands-on experience with Lakehouse architectures (Databricks, Snowflake, or similar)
• 7+ years of overall data engineering experience with large-scale distributed systems

Organization Overview 

Lilly IT builds and maintains capabilities using cutting edge technologies like most prominent tech companies. What differentiates Lilly IT is that we redefine what’s possible through tech to advance our purpose – creating medicines that make life better for people around the world, like data driven drug discovery and connected clinical trials. We hire the best technology professionals from a variety of backgrounds, so they can bring an assortment of knowledge, skills, and diverse th