Intermediate Site Reliability Engineer, Datastores : Database Reliability

GitLab is an open core software company that develops the most comprehensive AI-powered DevSecOps Platform, used by more than 100,000 organizations. Our mission is to enable everyone to contribute to and co-create the software that powers our world. When everyone can contribute, consumers become contributors, significantly accelerating the rate of human progress. This mission is integral to our culture, influencing how we hire, build products, and lead our industry. We make this possible at GitLab by running our operations on our product and staying aligned with our values. Learn more about Life at GitLab.

An overview of this role

 

Site Reliability Engineers (SREs) are responsible for keeping all user-facing services and other GitLab production systems running smoothly. SREs are a blend of pragmatic operators and software craftspeople that apply sound engineering principles, operational discipline, and mature automation to our environments and the GitLab codebase. We specialize in systems, whether it be networking, the Linux kernel, or some more specific interest in scaling, algorithms, or distributed systems.

The Database Reliability Team's mission is to build, run and own the entire lifecycle of the PostgreSQL database engine for GitLab.com. The team is focused on owning the reliability, scalability, performance & security of the database engine and its supporting services. The team should be seeking to build their services on top of Reliability::Foundations services and cloud vendor managed products, where appropriate, to reduce complexity, improve efficiency and deliver new capabilities quicker. 

GitLab.com is a unique site and it brings unique challenges–it’s the biggest GitLab instance in existence. In fact, it’s one of the largest single-tenancy open-source SaaS sites on the internet. The experience of our team feeds back into other engineering groups within the company, as well as to GitLab customers running self-managed installations

Responsibilities

• Automating every operational task is a core requirement for this role. For example, package updates, configuration changes across all environments, creating tools for automatic provisioning of user facing services, etc.
• Responding to platform emergencies, alerts, and escalations from Customer Support.
• Ensure systems exist to manage software life-cycles (e.g. Operating Systems) with a minimum of manual effort.
• Develop a fully automated multi-environment observability stack based on the existing SaaS system, and extend it to predict capacity needs based on the usage patterns.
• Plan for new service roll-outs, expansion and capacity management of existing services, and work with users to optimise their resource consumption.

As an SRE you will:

• Work on database reliability and performance aspects for GitLab.com from within the SRE team as well as work on shipping solutions with the product.
• Analyze solutions and implement best practices for our main PostgreSQL database cluster and its components.
• Work on observability of relevant database metrics and make sure we reach our database objectives.
• Work with peer SREs to roll out changes to our production environment and help mitigate database-related production incidents.
• OnCall support on rotation with the team.
• Provide database expertise to engineering teams (for example through reviews of database migrations, queries and performance optimizations).
• Work on automation of database infrastructure and help engineering succeed by providing self-service tools.
• Use the GitLab product to run GitLab.com as a first resort and improve the product as much as possible.
• Plan the growth of GitLab's database infrastructure.
• Design, build and maintain core database infrastructure pieces that allow GitLab to scale to support hundreds of thousands of concurrent users.
• Support and debug database production issues across services and levels of the stack.
• Make monitoring and alerting alert on symptoms and not on outages.
• Document every action so your learnings turn into repeatable actions and then into automation.

You may be a fit to this role if you:

• Have strong