Jobs and Careers
NE
Engineering Manager, Big Data Analytics Platform
NetflixLos Gatos, United Statesfull_timeVerifiedPosted 16 Apr 2024
💰 $190,000/yr
About the role
Netflix is the world’s leading streaming entertainment provider with over 260 million paid subscribers in over 190 countries around the world. This success has depended on an industry-leading engineering organization building cloud-native services with exceptional scale and resilience.
Big Data Analytics Platform (BDAP) team is responsible for providing orchestration and compute technologies to ETL and to access nearly 1 ExaByte of data in the Netflix Warehouse. It is also responsible for the underlying table format, Iceberg (an industry standard table format and an open source project that originated in this team), metadata, security and the life-cycle of the data (janitoring, compacting etc). Two Engineering Manager roles are open in the BDAP team.
Engineering Manager, Big Data OrchestrationThe Big Data Orchestration (BDO) team provides an opinionated, cost-efficient, reliable, easy-to-use orchestration platform to meet Netflix’s analytics and machine learning needs. BDO enables scheduling, orchestrating, and executing big data workflows and jobs. The core of Orchestration is Maestro, the next-generation Data Workflow Orchestration platform to meet Netflix's current and future needs. It is a general-purpose workflow orchestrator that provides a fully managed workflow-as-a-service (WAAS). The team built Maestro from the ground up, enabling high throughput, horizontal scalability, and advanced parametrized, event-based scheduling.
The BDO team is building an ETL framework for incremental data processing for increased data accuracy, freshness, and easy backfill to directly impact the cost and operational efficiency of Analytics and ML. Orchestration is powered by the federated job execution engine, Genie under the hood, which abstracts away query engine specifics from users. The suite of Orchestration products includes sophisticated error classification and intelligent job auto-remediation using Pensive and Nightingale.
Engineering Manager, Big Data Compute SparkThe Big Data Compute Spark (BDCS) team provides high-performance, scalable, distributed data processing at rest with the customized-for-Netflix fork of Apache Spark on Hadoop to meet Netflix’s large scale data analytics and machine learning needs. This team of 7 people (and growing) is central to batch data processing and analytics in the Data Platform at Netflix. It provides support for Spark to ETL data into the Exabytes-scale data warehouse and a variety of ways (for casual and power users) to access that data using Spark.The team is working on integrating Spark accelerators to further drive performance, evaluating containerization platforms to run Spark, building better batch and real time observability into Spark workflows and no/low touch ways to use Spark. This is a dream team of highly passionate and intelligent engineers who work well together. The team works on solving challenging problems at scale that have a huge impact on the Data Platform. The team has PMC members and committers who shape and contribute to open-source projects.
Check out some of our talks on Spark.
Big Data Analytics Platform (BDAP) team is responsible for providing orchestration and compute technologies to ETL and to access nearly 1 ExaByte of data in the Netflix Warehouse. It is also responsible for the underlying table format, Iceberg (an industry standard table format and an open source project that originated in this team), metadata, security and the life-cycle of the data (janitoring, compacting etc). Two Engineering Manager roles are open in the BDAP team.
Engineering Manager, Big Data OrchestrationThe Big Data Orchestration (BDO) team provides an opinionated, cost-efficient, reliable, easy-to-use orchestration platform to meet Netflix’s analytics and machine learning needs. BDO enables scheduling, orchestrating, and executing big data workflows and jobs. The core of Orchestration is Maestro, the next-generation Data Workflow Orchestration platform to meet Netflix's current and future needs. It is a general-purpose workflow orchestrator that provides a fully managed workflow-as-a-service (WAAS). The team built Maestro from the ground up, enabling high throughput, horizontal scalability, and advanced parametrized, event-based scheduling.
The BDO team is building an ETL framework for incremental data processing for increased data accuracy, freshness, and easy backfill to directly impact the cost and operational efficiency of Analytics and ML. Orchestration is powered by the federated job execution engine, Genie under the hood, which abstracts away query engine specifics from users. The suite of Orchestration products includes sophisticated error classification and intelligent job auto-remediation using Pensive and Nightingale.
Engineering Manager, Big Data Compute SparkThe Big Data Compute Spark (BDCS) team provides high-performance, scalable, distributed data processing at rest with the customized-for-Netflix fork of Apache Spark on Hadoop to meet Netflix’s large scale data analytics and machine learning needs. This team of 7 people (and growing) is central to batch data processing and analytics in the Data Platform at Netflix. It provides support for Spark to ETL data into the Exabytes-scale data warehouse and a variety of ways (for casual and power users) to access that data using Spark.The team is working on integrating Spark accelerators to further drive performance, evaluating containerization platforms to run Spark, building better batch and real time observability into Spark workflows and no/low touch ways to use Spark. This is a dream team of highly passionate and intelligent engineers who work well together. The team works on solving challenging problems at scale that have a huge impact on the Data Platform. The team has PMC members and committers who shape and contribute to open-source projects.
Check out some of our talks on Spark.
What you will do:
- Partner to deliver the vision, strategy, and adoption of current and future technologies
- Form trusting cross-functional partnerships to align many engineering teams and ensure our solutions meet their needs, selflessly prioritizing work beyond the scope of your own domain
- Build, scale, and grow a team of outstanding engineers, challenge them to bring their best selves to work every day, and deliver industry-leading results in stability, performance, and efficiency
- Balance smart risks, investment in foundational technical work, paying off tech debt, and incremental improvements to deliver timely results across multiple critical strategies
What we are looking for:
- An experienced engineering leader who can drive a team of amazing and geographically distributed engineers to do their best work while building and maintaining strong partnerships with peers and stakeholders
- Ability to dive deep as needed to facilitate technical strategy trade-offs and zoom out to understand the big picture to shape the product
- Experience building and running extremely reliable platform systems supporting high-scale and critical workloads, including running a 24x7 on-call, 99.95+% availability, fast response and short mean-time-to-restore se
Apply for this role
Generate a tailored application kit with a matched cover letter, interview prep, and CV highlights — in under 60 seconds.
Apply Now →Generate Application KitFree account required — sign up in 30s