Principal Software Engineer - Ad Platform

Job Posting Title:

Principal Software Engineer - Ad Platform

Req ID:

10110229

Job Description:

On any given day at Disney Entertainment & ESPN Technology, we’re reimagining ways to create magical viewing experiences for the world’s most beloved stories while also transforming Disney’s media business for the future. Whether that’s evolving our streaming and digital products in new and immersive ways, powering worldwide advertising and distribution to maximize flexibility and efficiency, or delivering Disney’s unmatched entertainment and sports content, every day is a moment to make a difference to partners and to hundreds of millions of people around the world.

A few reasons why we think you’d love working for Disney Entertainment & ESPN Technology

• Building the future of Disney’s media business: DE&E Technologists are designing and building the infrastructure that will power Disney’s media, advertising, and distribution businesses for years to come.

• Reach & Scale: The products and platforms this group builds and operates delight millions of consumers every minute of every day – from Disney+ and Hulu, to ABC News and Entertainment, to ESPN and ESPN+, and much more.

• Innovation: We develop and execute groundbreaking products and techniques that shape industry norms and enhance how audiences experience sports, entertainment & news.

The products and platforms the Ad Platforms group builds and operates delight millions of consumers every minute of every day – from Disney+ and Hulu, to ABC News and Entertainment, to ESPN and ESPN+, and much more.  We develop and execute groundbreaking products and techniques that shape industry norms and enhance how audiences experience sports, entertainment & news.

The Ad Platforms team is responsible for building and operating the technology that supports Disney’s industry-leading, multi-billion dollar global advertising business. Working across all of Disney’s streaming brands and linear channels, this team’s scope is wide ranging - they impact everything from core infrastructure development that powers Disney’s proprietary ad server, to reimagining viewer-first ad experiences and formats. This team’s award-winning, in-house built technologies are the backbone of Disney’s ad business and a critical element of the Company’s streaming future.

The Disney Ad Platforms Data team is at the forefront of building a modern advertising data platform to support our organization’s data, analytics and business intelligence needs. Our mission is to enable data engineers, data scientists, analysts, and decision-makers to derive value from our data ecosystem.

Job Summary:

We are seeking a Principal Software Engineer with extensive experience in the Databricks platform and advanced big data engineering practices. This role involves architecting and implementing scalable, secure, and high-performing data solutions to power the Ad Platforms Data platform and support mission-critical data initiatives, including real-time and batch processing pipelines.

As a hands-on technical leader, you will design data architectures that scale to handle terabytes-per-hour throughput, ensuring optimal performance, compliance, and alignment with business needs. You will play a pivotal role in guiding teams, collaborating across departments, and driving innovation. 

Responsibilities:

Technical Leadership and Architecture

• Architect and implement highly scalable, secure, and efficient ETL/ELT pipelines for both real-time and batch data processing.

• Design complex, multi-source data ecosystems and ensure seamless integration with tools like Databricks, Delta Lake, and Snowflake.

• Optimize data pipelines and workflows for performance, cost, and reliability, leveraging cutting-edge tools like Airflow, Spark, Flink, and Databricks Delta Sharing.

• Lead the adoption of best practices in data warehousing, lakehouse architecture, and metadata management.

• Champion data security, privacy, and governance, implementing frameworks for lineage tracking, cataloging, and compliance (e.g., SOx, GDPR, CCPA).

• Drive implementation of Unity Catalog for data governance and cross-account querying.

Platform Development and Operational Excellence

• Able to assist in the set up and manage AWS environments, including S3, IAM Roles, Network Peering, and other infrastructure components.

• Oversee Airflow integration with Databricks Spark clusters, manage user access, and implement robust secret management solutions.

• Lead the CI/CD process with gated code releases, ensuring code quality and scalable deployment processes.

• Implement inno