Senior Site Reliability Engineer, HPC Infrastructure and Platforms

Requisition Id 11976 

Overview:  

Are you looking for a way to use your hard-earned SRE skills in a more ambitious environment where you can also help protect national security? The National Center for Computational Sciences (NCCS) at Oak Ridge National Lab (ORNL), which hosts several of the world’s most powerful computer systems, is seeking highly qualified individuals to play a key role in improving the security, performance, and reliability of the NCCS computing infrastructure which supports multiple highly ranked Top500 Supercomputers, including the world’s first exaflop system, Frontier!  

  

The Team:  

As a Senior Site Reliability Engineer, you will work within the HPC Infrastructure and Platforms group to support all activities of our supercomputer center. Our primary platform is the OLCF Slate Service, built on Kubernetes and Red Hat OpenShift, which provides a container orchestration service for running critical operation applications and user-managed persistent applications that run alongside our OLCF Supercomputer systems and other OLCF-run HPC clusters.  

  

Major Duties/Responsibilities:  

• Lead ongoing improvements in reliability and scalability for our Kubernetes and Linux based applications and services.  

• Contribute as senior technical resource to deliver highest-quality practices and standards for the center.  

• Provide primary operational support and engineering for production applications.  

• Define and implement define important metrics, processes and drive continuous improvement.  

• Influence the architecture and implementation of solutions.  

• Tune operating systems and applications to increase performance and reliability of services.  

• Mentor junior staff and enable them for success.  

• Diagnose system operational problems quickly and optimally.  

• Participate in on-call rotation providing 24-hour, 7-day support and off-hours maintenance windows.  

• Coordinate with vendors to resolve hardware and software problems.  

• Drive ORNL’s mission by aligning behaviors, priorities, and interactions with our core values of Impact, Integrity, Teamwork, Safety, and Service. Promote diversity, equity, inclusion, and accessibility by encouraging a respectful workplace – in how we treat one another, work together, and measure success.  

  

Basic Qualifications:  

• Bachelor’s Degree in computer science or closely related field and at least 8 years proven experience as an SRE/Systems Engineer. An equivalent combination of education and experience may be considered.  

  

Preferred Qualifications:  

• Excellent interpersonal/communication skills, and desire to work as part of a team.  

• Solid understanding of Unix system fundamentals and common network protocols.  

• Experience managing Linux/UNIX operating systems in a heterogeneous environment.  

• Proficiency in networked computing environment concepts.  

• Ability to develop and maintain programs and scripts that aid in the operation and automation using various shell (primarily bash) and high-level languages (Python or Go).  

• Ability to proactively identify performance issues, problems, and areas for improvement.  

• Ability to identify requirements and to define, plan, and implement requisite solutions.  

• Ability to plan, organize, prioritize tasks, and complete assigned projects with minimal direction.  

• Experience with continuous integration and continuous deployment software methodologies and how they apply to SRE/systems engineering.  

• An understanding of code review and familiarity with tools like GitHub and GitLab  

• Experience using tools such as Nagios, Grafana and Prometheus to supervise systems, metrics, and build dashboards.  

• Experience crafting and implementing highly available systems/services using virtual machines and Kubernetes resources.  

• Experience participating in an opensource community with patches accepted upstream.  

• Experience deploying and maintaining automated configuration management software such as Puppet or Ansible  

• Experience implementing systems-level security technologies like SELinux and following standard processes in security.  

  

Special Requirement:  

This position requires the ability to obtain and maintain a clearance from the Department of Energy. As such, this position is a Workplace Substance Abuse program (WSAP) testing designed position which requires passing a pre-