Senior Storage and Data Production Engineer

Production engineering is a team that involves designing, building, and maintaining large-scale production systems with high efficiency and availability. It encompasses various areas, including software and systems engineering practices, storage, data management, and services. Production Engineers possess expertise in different domains, such as storage architecture, high-performance distributed storage, data management, systems, networking, coding, database management, capacity planning, continuous delivery, and deployment, as well as open-source cloud-enabling technologies like Kubernetes, containers, and virtualization. Their responsibilities include ensuring reliable, scalable, high-performance storage solutions, optimizing data placement and access patterns, managing large-scale distributed storage systems, and ensuring low-latency data access for high-performance computing (HPC) and AI/ML workloads.

Production Engineers at NVIDIA ensure that our internal and external-facing GPU cloud services have reliability and uptime as promised to the users while enabling developers to make changes to the existing system through careful preparation and planning while keeping an eye on capacity, latency, and performance. This role also requires an approach focused on automating storage operations, improving data access efficiency, and optimizing storage performance. Much of our software development focuses on eliminating manual work through automation, performance tuning, and growing the efficiency of storage and production systems.

What You Will Be Doing:

• Design, implement, and support large-scale storage clusters, ensuring scalability, high availability, and data integrity.

• Develop and maintain storage monitoring, logging, and alerting systems to ensure proactive detection and resolution of performance issues.

• Work with AI/ML workloads to optimize storage architectures for low-latency access, efficient caching, and high-throughput performance. Improve the lifecycle of storage services – from inception and design to deployment, operation, and continuous optimization.

• Support storage services before they launch through activities such as system design consulting, developing automation frameworks, capacity management, and launch reviews.

• Maintain storage infrastructure once live by monitoring availability, latency, and system health, using predictive analytics and AI-driven automation.

• Optimize storage efficiency through compression, duplication, tiering strategies, and intelligent workload placement.

• Scale storage systems sustainably using AI/ML-driven automation, policy-based tiering, and dynamic data migration techniques. Ensure data security and compliance by implementing encryption, access controls, and auditing mechanisms for storage systems.

• Practice sustainable incident response and blameless postmortems. Be part of an on-call rotation to support storage and production systems.

What We Need To See:

• BS degree or equivalent experience in Computer Science, Storage Systems, or a related technical field (e.g., physics, mathematics), and 5+ years of practical experience.

• Experience with high-performance storage solutions, including parallel file systems (Lustre, GPFS), distributed storage (Ceph, MinIO), and enterprise-scale object storage (S3, NetApp, Pure Storage, etc.).

• Solid understanding of block, file, and object storage technologies, including their performance characteristics and standard methodologies.

• Experience with storage networking protocols such as NFS, SMB, iSCSI, Fibre Channel, RDMA, and NVMe over Fabrics.

• Expertise in algorithms, data structures, complexity analysis, software design, and maintaining large-scale Linux-based storage systems.

• Experience in one or more of the following: C/C++, Java, Python, Go, Perl, or Ruby for storage automation, monitoring, and performance tuning.

• Hands-on experience with infrastructure configuration management tools like Ansible, Chef, Puppet, and Terraform for automating storage deployments.

• Experience with observability and tracing tools like InfluxDB, Prometheus, and the Elastic stack for monitoring storage system health.

Ways to stand out from the crowd:

• Deep understanding of large-scale distributed storage architectures, replication strategies, and erasure coding techniques. Proven experience in capacity planning, performance tuning, and troubleshooting high-throughput storage systems.

• Experience with Git, code review, pipelines, and CI/CD for handling infrastructure as code. Interest in analyzing and improving distributed storage system performance at scale. Strong debugging skills with a systematic pro