Senior Site Reliability Engineer, General

The GitLab DevSecOps platform empowers 100,000+ organizations to deliver software faster and more efficiently. We are one of the world’s largest all-remote companies with 2,000+ team members and values that foster a culture where people embrace the belief that everyone can contribute. Learn more about Life at GitLab.

An overview of this role

Site Reliability Engineers (SREs) are responsible for keeping all user-facing services and other GitLab production systems running smoothly. SREs are a blend of pragmatic operators and software craftspeople that apply sound engineering principles, operational discipline, and mature automation to our environments and the GitLab codebase. We specialize in systems, whether it be networking, the Linux kernel, or some more specific interest in scaling, algorithms, or distributed systems.

GitLab.com is a unique site and it brings unique challenges–it’s the biggest GitLab instance in existence. In fact, it’s one of the largest single-tenancy open-source SaaS sites on the internet. The experience of our team feeds back into other engineering groups within the company, as well as to GitLab customers running self-managed installations

• Automating every operational task is a core requirement for this role. For example, package updates, configuration changes across all environments, creating tools for automatic provisioning of user facing services, etc.
• Responding to platform emergencies, alerts, and escalations from Customer Support.
• Ensure systems exist to manage software life-cycles (e.g. Operating Systems) with a minimum of manual effort.
• Develop a fully automated multi-environment observability stack based on the existing SaaS system, and extend it to predict capacity needs based on the usage patterns.
• Plan for new service roll-outs, expansion and capacity management of existing services, and work with users to optimize their resource consumption.

Some examples of the projects you could work on:

• Work on the GitLab core projects such as, GitLab Rails, GitLab Workhorse, Gitaly, etc.
• Work on projects that create deployment packages of GitLab such as, GitLab Helm Chart, omnibus-gitlab, etc.
• Coding infrastructure automation with Chef and Terraform.
• Work on application side changes required for GitLab.com migration to Kubernetes
• Improving our Prometheus Monitoring or building new Metrics

What you’ll do

• Be on a PagerDuty rotation to respond to GitLab.com availability incidents and provide support for service engineers with customer incidents.
• Analyze existing, create and maintain new GitLab.com Service Level Objectives.
• Troubleshoot, evaluate and resolve operational challenges contributing to defined SLO's.
• Define, improve, and engage in adapting architectural application bottlenecks as observed on GitLab.com.
• Work with other engineering stakeholders on resolving larger architectural bottlenecks and participate by offering GitLab.com point of view.
• Work in close collaboration with software development teams to shape the future roadmap and establish strong operational readiness across teams.
• Scale systems through automation, improving change velocity and reliability.
• Leverage technical skills to partner with team members and be comfortable diving into a problem as needed.
• Work with counterparts in other teams of the Infrastructure department to improve infrastructure running with Chef, Terraform and Kubernetes.
• Make monitoring and alerting alert on symptoms and not on outages.
• Document every action so your findings turn into repeatable actions–and then into automation.
• Debug production issues across services and levels of the stack.

What you’ll bring 

• Have programming skills - Preferably wit