Software Engineer L4, LLM Evaluation & Infrastructure, Machine Learning Platform

Netflix is one of the world's leading entertainment services, with over 300 million paid memberships in over 190 countries enjoying TV series, films and games across a wide variety of genres and languages. Members can play, pause and resume watching as much as they want, anytime, anywhere, and can change their plans at any time.

Machine Learning/Artificial Intelligence powers innovation in all areas of the business, from helping members choose the right title for them through personalization, to better understanding our audience and our content slate, to optimizing our payment processing and other revenue-focused initiatives. Building highly scalable and differentiated ML infrastructure is key to accelerating this innovation.

The Opportunity

We are looking for a driven Software Engineer (L4/L5) to join our Machine Learning Platform (MLP) org. MLP’s charter is to maximize the business impact of all ML use cases at Netflix through highly reliable and flexible ML tooling and infrastructure that support personalization, studio algorithms, virtual production, growth intelligence, and content understanding. In this role, you will design and operate the systems that measure LLM quality, safety, and performance at scale—closing the loop from model development to production through rigorous, reproducible evaluation. 

In this role you will get to: 

• Build the evaluation platform that runs large-scale LLM eval suites across modalities and tasks (e.g., content understanding, personalization prompts, assistant use cases), integrating with batch/online inference (including vLLM-based backends) and experiment tracking to deliver reliable, reproducible metrics.  

• Operationalize benchmark coverage alongside Netflix-specific task suites and user-journey-grounded prompts; automate result collection, statistical analysis, and drift detection.  

• Develop high-quality synthetic data and labeling pipelines to expand coverage, reduce bias, and continuously refresh eval corpora; codify data provenance and sampling policies.  

• Partner deeply with model developers and platform teams to co-design APIs for submitting eval jobs, adding new tasks/metrics, and defining SLO-like quality thresholds that unblock launches while preventing regressions.  

• Contribute beyond evaluation across the GenAI/FM stack when needed:

  • Research workflows (orchestration, queueing/caching/failure isolation, artifact lineage, experiment management) that keep scientists productive at scale.

  • Inference foundations (vLLM/TGI-class serving, routing, safety filters, latency/throughput tuning, cost controls) and batch evaluation/inference at production scale.

  • Observability for the whole loop: dataset/version provenance, model/build metadata, metric lineage, and run reproducibility.

Minimum Job Qualifications

• Experience in ML engineering on production systems dealing with training or inference of deep learning models

• Proven track record of building and operating large-scale infrastructure for machine learning use cases

• Experience with cloud computing providers, preferably AWS

• Comfortable with ambiguity and working across multiple layers of the tech stack to execute on both 0-to-1 and 1-to-100 projects

• Adopt and promote best practices in operations, including observability, logging, reporting, and on-call processes to ensure engineering excellence.

• Excellent written and verbal communication skills

• Comfortable working in a team with peers and partners distributed across (US) geographies & time zones.

Preferred Qualifications

• End-to-end foundation-model lifecycle exposure: pre-train checks, post-train regression, and pre-launch gates—understanding where and how evaluation fits.

• Built or contributed to an evaluation platform at scale (batch/online evals, multi-modal tasks, queueing, caching, failure isolation) with strong SLIs/SLOs.

• Experience building evaluation data pipelines (synthetic generation, labeling, sampling) with provenance and governance

• Platform mindset: craft usable APIs/UX so modeling teams can submit tasks, compare runs, and gate launches with SLO-like thresholds. 

• Bonus signals across the broader stack: experience with reinforcement learning, agent modeling, AI alignment, distributed training, vector search/feature store