Observability - Intake Services - Principal Go Engineer

Elastic is an open source search company that powers enterprise search, observability, and security solutions built on one technology stack that can be deployed anywhere. From finding documents to monitoring infrastructure to hunting for threats, Elastic makes data usable in real time and at scale. Thousands of organizations worldwide, including Barclays, Cisco, eBay, Fairfax, ING, Goldman Sachs, Microsoft, The Mayo Clinic, NASA, The New York Times, Wikipedia, and Verizon, use Elastic to power mission-critical systems. Founded in 2012, Elastic is a distributed company with Elasticians around the globe. Learn more at elastic.co

About the role 

The Observability team is developing and providing solutions to application developers and engineers that run infrastructure and services supporting these applications. Elasticsearch is an efficient datastore for logs, metrics, application traces and profiling data. The Observability team builds and maintains solutions that make getting insights from this data turnkey and efficient, such as our APM, Infrastructure Monitoring, Logs and Universal Profiling solutions. When developing these solutions, we think about the problem end-to-end: how do we automatically collect data from common data sources, how do we store it efficiently in Elasticsearch, how do we present this information to the user, what actions do we take on the insights from the data? All of these aspects are important in bringing a turnkey solution to the market. 

As a Principal Go Engineer in the Observability Intake Services team, you will be part of a team developing high quality observability products that are essential to the ingest flow of the Elastic Observability solution. The team’s main focus is on building and maintaining backend intake service components for cloud managed and on-prem deployments, and owning operational stability through extensive monitoring and SLOs. We have a focus on quality, performance, reliability, security and testability, as well as automation and continuous improvements. 

We build components and services that receive and validate signals from various edge collectors, such as Elastic APM, OpenTelemetry SDKs, as well as profiling agents, and transform them into Elasticsearch documents. The intake service components control the amount of data flowing into Elasticsearch and can buffer data temporarily without adding overhead to the edge applications. 

We appreciate community contributions, and as such, engaging with the open source community is also an important part of this role.

You’ll be collaborating with immediate team colleagues and across team boundaries. The team is diverse and distributed across the world, and collaborates on a daily basis over GitHub, Zoom, and Slack. 

What you will be doing:

• Lead technical designs and implementation for the observability intake services in Go
• Be a hands-on contributor to the team’s services
• Maintain, update and improve existing code and dependencies while ensuring automated test coverage.
• Create and improve the development infrastructure using modern tooling such as Tilt and Terraform.
• Build a strong backend solution for the OpenTelemetry data flow.
• Design data formats for optimized data storage and query performance in Elasticsearch.
• Provide Pull Request reviews and strong technical guidance to your peers.
• Build SLIs and SLOs and participate in an on-call rotation for the services in the team’s ownership.
• Proactively participate in team conversations and contribute to a positive team culture. 
• Collaborate with other parts of the company on cross team features and topics.
• Collaborate with the support team on customer requests, and engage with the community via pull requests, issues and discuss forums. 

What you bring along: 

• A positive attitude and a curious mindset.
• Expert level knowledge of Go, including advanced topics such as memory management, benchmarking and profiling.
• Strong knowledge in systems design.
• Strong analytical, debugging, and troubleshooting skills.
• Experience developing high throughput, low latency services.
• Hands-on experience with Docker and Kubernetes. 
• Ability to work independently in a globally distributed team
• Willingness and ability to share knowledge with others and enable your peers.

Bonus points

• Experience and knowledge in the observability space, including OpenTelemetry
• Experience working with stream process