Senior Distributed Systems Engineer

Responsibilities

• Work with researchers to scale up the systems required for our next generation of models trained on multi-thousand GPU clusters.
• Profile and optimize our model training code-base to achieve best in class hardware efficiency.
• Build systems to distribute work across massive GPU clusters efficiently.
• Design and implement methods to robustly train models in the presence of hardware failures.
• Build tooling to help us better understand problems in our largest training jobs.

Experience

• 5+ years of work experience.
• Experience working with multi-modal ML pipelines, high performance computing and/or low level systems.
• Passion for diving deep into systems implementations and understanding their fundamentals in order to improve their performance and maintainability.
• Experience building stable and highly efficient distributed systems.
• Strong generalist Python and Software skills including significant experience with Pytorch.
• Good to have experience working with high performance C++ or CUDA.

Compensation

• The pay range for this position in California is $180,000 - $250,000yr; however, base pay offered may vary depending on job-related knowledge, skills, candidate location, and experience. We also offer competitive equity packages in the form of stock options and a comprehensive benefits plan.