Principal Data Engineer (Hadoop, Spark, Flink, Kafka)

We shaped the earliest forms of ad tech, and we’re looking for technical expertise to help shape its future. Our customers have unique problems that can only be solved at internet scale, and that’s where the technical skills of our team make a real difference. 

Our exchange handles over 350 billion requests every day (for comparison Google serves an estimated 9 billion searches a day), all running in our own global data centers. Every member of our technology team has an enormous amount of autonomy in building and managing our systems to support and enable our growing level of scale. Through the transparency of our technology, dedication to innovation and integrity, and long-standing customer relationships, we lead through change. 

What’s it like to work at Index? 

We have more than 550 Indexers around the globe dedicated to building a safe and transparent marketplace that provides a trusted experience for consumers. 

Index is an exciting and fast-paced place to work. We’re built on our values of change, support, learning and teaching, trust, and intention. We pride ourselves on our independence and openness, not only in our technology, but in our teams, too. Our diverse and inclusive culture celebrates how we can leverage our unique differences to help drive Index forward. 

Our culture of success is truly supportive and collaborative. In working together across our teams, we’re continually investing in the people and technology to solve the industry’s most complex problems. As we extend the promise of ad tech to every channel, we’re looking for talented engineers to help advance Index, and the industry, forward. 

Are you ready to join the programmatic evolution? 

Index Exchange funds the open web. Content and journalism across the internet are funded through advertising, and we are the engine that helps to make that happen transparently, safely and efficiently. Handling hundreds of billions of auctions per day within milliseconds requires an intense understanding of the exchange and the ecosystem that we live in. 

Our business is growing significantly every year and is poised to grow even faster. Our people and our platforms are the foundation and enabler of that growth. We are significantly expanding our technology teams, and are looking for technologists with a passion for high performance software development, and a drive to deliver software products and platforms that enable and empower industries at a global scale. 

About The Role 

Join our global organization at the forefront of innovation in the Ad-Tech industry.  We are seeking a dynamic and experienced Principal Engineer, Data Systems to lead across data domains and teams, driving strategic outcomes for our organization. The ideal candidate will be equal parts responsible for driving initiatives at deep technical level and ensuring robust operational rigor to achieve stability and scale in our data environments. 

Here’s What You’ll be Doing 

Leadership Across Domains: 

• Provide strategic leadership across multiple domains, collaborating with cross-functional teams such as product, operations, infrastructure, and platform engineering to achieve business objectives. 
• Foster a culture of innovation, continuous improvement, and collaboration within the data systems team. 
• Advocate for strategic directions that reduce our complexity and tech stack footprint. 

Technical Depth: 

• Demonstrate deep expertise in our core data streaming components, including Kafka, Spark, Hadoop, Presto/Trino, Flink and Redis in a low latency environment. 
• Architect relational data structures for performance, scale, and continuity. 
• Drive technical excellence by staying abreast of industry trends, evaluating emerging technologies, and making recommendations for adoption. 
• Lead initiatives with other cross-functional Principal engineering staff that are supporters and consumers of your data systems.

Operational Rigor: 

• Ensure robust operational processes are in place to guarantee stability and scalability of database environments. 
• Develop and implement best practices for database administration, monitoring, and maintenance using open-source observability and instrumentation tools. 
• Develop and enhance testing and production release procedures with a automation-first core principle to end-to-end delivery. 

Collaboration and Communication: 

• Collaborate effectively with cross-functional teams, including developers, system administrators, and data scientists, to optimize database performance and functionality. 
• Communicate complex technical concepts to non-technical stakeholders, fo