Associate Principal, Site Reliability Engineering

To be considered for this position, applications and resumes are accepted only through our careers site by directly applying to the posted job. We do not accept unsolicited resumes or sales solicitations from staffing agencies. Any OCC employee wishing to submit a referral must do so through their Workday account. Any resume submitted outside of an active job posting will not be considered for employment.

What You'll Do:

Provide strong support for the availability and performance of OCC’s next generation Ovation platform. Enhance system reliability and developer productivity through automation. Provide guidance to development, platform teams, in the areas of cloud technologies, application profiling and monitoring, logging, metrics collection and analysis.

Primary Duties and Responsibilities:

To perform this job successfully, an individual must be able to perform each primary duty satisfactorily.

• Collaborate with development, operations and infrastructure teams to ensure availability of services, and to work through implementation issues

• Develop automation for incident response and to prevent problem recurrence

• Create and enhance runbooks to respond to service outages or degradations

• Assess the production readiness of services

• Define and track operational metrics for production performance, reliability, scalability and availability

• Architect, develop and maintain shared services and tools to improve reliability and reduce toil across the organization

• Contribute to the team’s continuous improvement through research, retrospectives, discussion groups and code reviews

• Influences timelines and expectations amongst the team

• Provide knowledge by guiding and mentoring junior members, and preparing stories for the sprint backlog

Supervisory Responsibilities:

None

Qualifications:

The requirements listed are representative of the knowledge, skill, and/or ability required.  Reasonable accommodations may be made to enable individuals with disabilities to perform the primary functions.

• [Required] Experience with maintaining and troubleshooting large-scale distributed systems

• [Required] Experience with Agile / Scrum methodology

• [Required] Able to succeed in fast-paced environment with frequent changes

• [Required] Comfortable communicating with both technical and non-technical audiences

• [Required] Strong documentation skills

• [Required] Analytical problem-solving approach

• [Required] Self-starter – takes the initiative to research, learn and deliver. Anticipates the play

• [Required] Team player – humble, collaborative, and focused on making sure the entire team succeeds

Technical Skills:

• [Required] Experience managing infrastructure in public cloud environments like AWS (preferred), Azure or GCP

• [Required] Experience with AIOps and predictive analysis for anomaly detection, forecasting system capacity using monitoring and alerting tools like Splunk, AppDynamics, Datadog, StackDriver, Sysdig, Prometheus or Grafana

• [Required] Programming/scripting experience in languages like Java, Bash, Python or Go

• [Required] Experience with distributed messaging systems like Kafka, RabbitMQ, or ActiveMQ

• [Required] Experience with container orchestration systems like Kubernetes, Mesos, Docker Swarm or Rancher

• [Required] Experience with using Continuous Integration and Continuous Delivery (CI/CD) tools like Jenkins, Travis, Harness, Appveyor, CodeBuild or CodePipeline

• [Required] Familiarity with leveraging large language models (LLMs) to automate and optimize SRE workflows. This may include using AI-powered tools to perform tasks such as, writing scripts, summarizing incident reports, or even creating and maintaining AI workloads.

• [Required] Familiarity with leveraging large language models (LLMs) to automate and optimize SRE workflows. This may include using AI-powered tools to perform tasks such as, writing scripts, summarizing incident reports, data analysis or even creating and maintaining AI workloads.

• [Required] Basic exposure to Chaos Engineering tools like, Gremlin, Chaos Monkey, Harness Chaos Engineering, or cloud-native fault injection services like AWS FIS.

Education and/or Experience:

• [Required] Bachelor’s or Master’s Degrees in Computer Science, Information Systems or other related field, or equivalent work experience

• [Required] Minimum of 4+ years of experience in Site Reliability Engineering / D