Lead Data Engineer

Job Posting Title:

Lead Data Engineer

Req ID:

10136508

Job Description:

Technology is at the heart of Disney’s past, present, and future. Disney Entertainment and ESPN Product & Technology is a global organization of engineers, product developers, designers, technologists, data scientists, and more – all working to build and advance the technological backbone for Disney’s media business globally.

The team marries technology with creativity to build world-class products, enhance storytelling, and drive velocity, innovation, and scalability for our businesses. We are Storytellers and Innovators. Creators and Builders. Entertainers and Engineers. We work with every part of The Walt Disney Company’s media portfolio to advance the technological foundation and consumer media touch points serving millions of people around the world. 

Here are a few reasons why we think you’d love working here:

Building the future of Disney’s media: Our Technologists are designing and building the products and platforms that will power our media, advertising, and distribution businesses for years to come. Reach, Scale & Impact: More than ever, Disney’s technology and products serve as a signature doorway for fans' connections with the company’s brands and stories. Disney+. Hulu. ESPN. ABC. ABC News…and many more. These products and brands – and the unmatched stories, storytellers, and events they carry – matter to millions of people globally.  Innovation: We develop and implement groundbreaking products and techniques that shape industry norms, and solve complex and distinctive technical problems.

Product Engineering is a unified team responsible for the engineering of Disney Entertainment & ESPN digital and streaming products and platforms. This includes product engineering, media engineering, quality assurance, engineering behind personalization, commerce, lifecycle, and identity.

Job Summary:

The Search Machine Learning (ML) team powers the ML aspects of search engines for Disney+ platform, in a highly collaborative environment. ML-based services are embedded inside the entire life cycle of search journey, from query understanding, semantic retrieval, and engagement-based re-ranking of contents. Working closely with our product stakeholders, we are constantly testing new ideas, and bringing the proven ones into fruition, through experimentation.

In this role, you will be leading the data operation aspects of our ML services, from data warehousing, building and owning ETL pipelines, and standing up feature stores for low-latency feature access to our ML micro-services. This is a highly collaborative role, where you will be able to own the entire lifecycle of data pipelines, communicate your requirements with upstream DATA teams, bring your data engineering skills and ideas into life by engineering efficient data pipelines, and help supporting the feature requirements by the ML engineers/data scientists.

Responsibilities and Duties of the Role:

• Design, own, and optimize our offline (Spark) and near-line (Kinesis, Kafka) data pipelines.
• Stand up scalable and performant online feature stores; build SDKs around both customized and provisioned feature stores.
• Integrate online feature stores with the ML service layer to enable real-time model inference and data serving.
• Scope data lineage dependencies and participate in cross-team data model design for upstream and downstream systems.
• Mentor junior data engineers on best practices for testing, validation, and deployment of ETL jobs, including pipeline and feature set versioning, and promoting code from non-prod to prod environments.

Required Education, Experience/Skills/Training:

Basic Qualifications

• 7+ years of data engineering experience, with 2+ years of relevant Feature Store experience
• Experience with design trade-offs for low-latency online feature stores
• Hands-on experience with sagemaker/Redis or any other equivalent online Feature Stores
• Experience with large-scale distributed data processing systems, such as Spark, Hadoop, Ray

Preferred Qualifications

• Familiarity with vector databases such as Faiss, Pinecone, Milvus, mongoDB, Qdrant
• Experience building streaming pipelines using Kafka, Kinesis

Required Education  

• Bachelor's (MS or PhD preferred) in Computer Science, Software Engineering, or a related field

The hiring range for this position in Santa Monica, CA is $155,700 - $208,700 per year and in New York City is $163,100 - $218,700 per year and in San Francisco, CA is $170,500 - $228,600 per year. The base pay actually offered will take into