Senior Linux HPC Systems Engineer

Requisition Id 12859 

 

Due to the security clearance requirements of this position, we are unable to consider EAD and Green Card holders. Additionally, Visa sponsorship of any kind is unavailable for this position (H1B, F1, OPT, H4, J1, etc.).

 

Overview:  

We are seeking a Senior Linux HPC Systems Engineer to design, operate and maintain clusters, servers, workstations that support services where science happens at ORNL! This position resides in the Emerging Technologies & Computing Group in the Information Technology Services Directorate at Oak Ridge National Laboratory (ORNL).

 

The Emerging Technology Computational Group facilitates ORNL goals through HPC systems engineering, integration, and support for the research community at ORNL. By providing design, deployment, optimization, monitoring, and tooling support across multiple clustered infrastructures, we facilitate Lab-wide R&D projects. Our HPC clusters range in scope from just a handful of nodes to more than fifty thousand cores.

 

Major Duties/Responsibilities: 

Our primary goal is to partner with research organizations to enable research excellence and delivery at ORNL. We work with other clustered computing and HPC groups to help research programs identify the best solutions for their needs. When we build our customer's environments, our team collaborates to design, implement, and maintain the systems from inception to retirement.

 

• Advocate and promote HPC and clustered computing services to researchers who process large data sets and/or develop code as a part of their project.
• Ensure the availability, performance, scalability, and security of production systems.
• Leverage automation and monitoring solutions that minimize our day-to-day maintenance and scout opportunities to optimize system management practices or system performance.
• Collaborate with technical POCs for the programs that we support to install and help tune the performance of various scientific toolsets.
• The Emerging Technologies & Computing Group optimizes workflows and monitoring solutions to take advantage of our 24/7 operations staff, which significantly reduces the need for off-hours support. We use Email, Jira, Confluence, Teams, Slack, and other collaboration solutions to stay in contact.
• Deliver ORNL’s mission by aligning behaviors, priorities, and interactions with our core values of Impact, Integrity, Teamwork, Safety, and Service. Promote diversity, equity, inclusion, and accessibility by fostering a respectful workplace – in how we treat one another, work together, and measure success.

 

Basic Qualifications:

• A Bachelor's degree in Computer Science or a related field of study with at least 8 years of proven experience is required, or an overall combination of equivalent experience may be considered.
• 5+ years’ managing UNIX/Linux Systems.
• 3+ years utilizing configuration management and automation tools such as Git, Jenkins, Ansible, or Puppet.
• Moderate proficiency in at least one scripting language such as Bash, Python, or equivalent.
• Experience performing advanced troubleshooting and system administration with Linux Servers.
• Experience supporting large data systems.
• A strong desire to push the envelope and identify new technologies and opportunities and be able to communicate the potential benefits of those choices to others within the team and our research partners.
• A collaborative and energetic mindset to thrive on the opportunity to build trust and credibility, and ultimately become a trusted advisor to our research teams.

 

Preferred Qualifications:

• Solid understanding of multiple operating systems and cluster technologies.
• Experience with Centos/RHEL, Ubuntu, VMware
• Understanding of HPC platforms to support users with SLURM job submissions and troubleshooting.
• Experience building and running containerized applications in an HPC environment.
• Experience with multiple deployment mechanisms like Diskless, Warewulf, and traditional deployment (cobbler, PXEboot, and/or Bright).
• Experience managing systems utilizing GPU (NVIDIA and AMD) clusters for AI/ML and/or image processing.
• Knowledge of networking fundamentals including TCP/IP, traffic analysis, common protocols, and network diagnostics.
• Experience with Infiniband networks and diagnostics.
• Extensive experience with High Performance Parallel File Systems (Lustre, WEKA, GPFS, etc).
• Experience with performance and diagnostic tools for benchmarking, analysis and tuning of systems, networking, and storage.
• Experience with Grafana, CheckMK, Nagios, Zabbix, SolarWinds, Ganglia, or other network and devi