Engineering Manager, Hardware Health

About the Team

OpenAI’s Hardware Health team is dedicated to ensuring the optimal performance and reliability of our custom-built hyperscale supercomputers. We focus on maximizing supercomputing capacity for research and ensuring that our researchers are minimally impacted by hardware faults. This team is critical in maintaining the infrastructure that supports cutting-edge AI research at OpenAI.

The Hardware Health team operates within the broader Platform organization, which is incubated inside OpenAI’s Research team. Our work is on the front lines of innovation, supporting the engineering and research required to train large-scale AI models of unprecedented capability.

About the Role

As an Engineering Manager on the Hardware Health team, you will lead a group of engineers responsible for ensuring the reliability, performance, and scalability of our supercomputing infrastructure. Your team will build infrastructure and automation to manage the lifecycle and operations of our cutting-edge hyperscale supercomputers. This includes monitoring, detecting, diagnosing, and resolving issues, as well as operationalizing scalable solutions.

Our team heavily uses Python and some Rust, and we are looking for someone who is excited to work in this stack and learn it if they're not familiar.

This role is based in San Francisco, CA. We use a hybrid work model of 3 days in the office per week and offer relocation assistance to new employees.

In this role, you will:

• Lead a team of software engineers to manage and optimize the critical infrastructure of our hyperscale supercomputers.

• Develop and implement strategies to monitor and maintain system health, ensuring minimal downtime and optimal performance.

• Collaborate closely with hardware engineers, systems engineers, and researchers to understand the infrastructure requirements and challenges of our AI training workloads.

• Drive the development of tools and automation to detect, diagnose, and mitigate hardware health-related issues.

• Build a team culture that prioritizes reliability, scalability, and performance, while fostering innovation and continuous learning.

• Create a diverse, equitable, and inclusive environment that encourages team members to contribute their best ideas and challenge assumptions.

You might thrive in this role if you:

• Have 5+ years of experience in engineering management, with a focus on large-scale infrastructure roles.

• Possess deep expertise in managing and optimizing hardware and software in large-scale, high-performance computing environments.

• Are a seasoned technical leader, excited to do hand-on technical work but equally excited to lead technical teams to peak performance.

• Have a strong background in troubleshooting complex system issues and developing solutions to prevent future occurrences.

• Bonus: Familiarity with modern AI infrastructure technologies (e.g., A100s, H100s, etc.), or experience working in large-scale HPC environments is a plus.

• Are skilled in collaborating across teams to solve complex problems and deliver high-impact results.

• Care deeply about diversity, equity, and inclusion, and have a track record of building inclusive teams.

• Have excellent communication skills, with the ability to convey complex technical concepts to both technical and non-technical audiences.

• Have a humble attitude, an eagerness to help your colleagues, and a desire to do whatever it takes to make the team succeed.

• Own problems end-to-end, and are willing to pick up whatever knowledge you're missing to get the job done.

About OpenAI

OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity. We push the boundaries of the capabilities of AI systems and seek to safely deploy them to the world through our products. AI is an extremely powerful tool that must be created with safety and human needs at its core, and to achieve our mission, we must encompass and value the many different perspectives, voices, and experiences that form the full spectrum of humanity. 

We are an equal opportunity employer and do not discriminate on the basis of race, religion, national origin, gender, sexual orientation, age, veteran status, disability or any other legally protected status. 

OpenAI Affirmative Action and Equal Employment Opportunity Policy Statement

For US Based Candidates: Pursuant to the San Francisco Fair Chance Ordinance, we will consider qua