Senior Storage System Software Developer

Company Description

Join us and make YOUR mark on the World!

Lawrence Livermore National Laboratory (LLNL) has turned bold ideas into world-changing impact advancing science and technology to strengthen U.S. security and promote global stability. 

Our mission spans four critical national security areas nuclear deterrence, threat preparedness, energy security, and multi-domain defense empowering teams to take on the toughest challenges of today and tomorrow. With a culture built on innovation and operational excellence, LLNL is a place where your expertise can make a real impact.

Job Description

We have an opening for a Senior Storage System Software Developer on a team that researches, designs, develops, maintains, and integrates software and hardware solutions that underpin scalable storage services within the Livermore Computing high-performance computing center. In this role, you will apply software development experience and broad systems-level mastery to support production parallel file systems and archival storage systems: to include troubleshooting, debugging, assistance to system administration staff to isolate software defects on production systems, and independent software development to address challenging issues on large-scale systems. Additionally, this position includes opportunities to implement new software features in archival storage and file systems such as High-Performance Storage System (HPSS), Lustre, and ZFS. This position is in the Livermore Computing Division within the Computing Principal Associate Directorate. 

This position offers a hybrid schedule, blending in-person and virtual presence. You will have the flexibility to work from home up to two days per week. 

This position will be filled at the SES.3/SES.4 level and will be filled at either level based on knowledge and related experience as assessed by the hiring team. Additional job responsibilities (outlined below) will be assigned if hired at the higher level.

You will

• Provide software engineering support for production file systems and/or long-term archival storage systems running at petabyte and billion-object scale. 
• Troubleshoot and debug highly scalable software-defined storage systems such as Lustre and/or HPSS. 
• Contribute to long-term maintenance of HPSS and/or Lustre codebases along with related projects (ZFS, Lustre Monitoring Tools, storage quota systems, etc.). 
• Design, implement, and maintain new features and performance improvements for HPSS and/or Lustre (and related projects). 
• Review colleagues’ code changes and integrate upstream patches into local versions of Lustre and/or HPSS codebases. 
• Develop and refine storage system monitoring applications. 
• Collaborate with cross-functional teams and across organizations to implement innovative solutions and/or resolve system-wide performance degradations and functionality defects in production storage systems. 
• Perform other duties as assigned. 

Additionally at the SES.4 level, you will 

• Serve as a technical subject matter expert and provide technical leadership for complex storage software and systems efforts.
• Mentor and develop technical staff across the organization and share expertise broadly with the next generation of storage professionals. 
• Lead cross-functional efforts to diagnose and resolve critical system-wide performance, scalability, and reliability issues.
• Identify and integrate innovative approaches using new technologies, articulating alternative solutions and their impacts. 
• Provide strategic technical guidance to project stakeholders, management and partner organizations. 

Qualifications

• Ability to maintain a U.S. DOE Q-level security clearance which requires U.S. citizenship. 
• Bachelor’s degree in computer science or related field or the equivalent combination of education and related experience. 
• Significant experience with file system internals and/or with hierarchical storage system concepts and systems — including tiered storage systems that integrate flash, HDD, and tape — used to implement long-term archival storage systems. 
• Significant experience in a production high performance computing environment. Experience operating storage systems in a production high performance computing (HPC) environment where unplanned downtime has significant operational consequences and end-user impact. 
• Advanced proficiency developing software in a team environment with two or more of the following programming languages: C, C++, Rust, or Python. 
• Proficiency in Linux command line environments. 
• Proficiency with distributed version control software (example: git). 
• Advanced verbal and written communication skills necessary to ef