Senior Software Engineer, Observability

CoreWeave is the AI Hyperscaler™, delivering a cloud platform of cutting edge services powering the next wave of AI. Our technology provides enterprises and leading AI labs with the most performant, efficient and resilient solutions for accelerated computing. Since 2017, CoreWeave has operated a growing footprint of data centers covering every region of the US and across Europe. CoreWeave was ranked as one of the TIME100 most influential companies of 2024.

As the leader in the industry, we thrive in an environment where adaptability and resilience are key. Our culture offers career-defining opportunities for those who excel amid change and challenge. If you’re someone who thrives in a dynamic environment, enjoys solving complex problems, and is eager to make a significant impact, CoreWeave is the place for you. Join us, and be part of a team solving some of the most exciting challenges in the industry.  

CoreWeave powers the creation and delivery of the intelligence that drives innovation. 

About the role

The Observability department plays a pivotal role in CoreWeave’s ability to bring the best performing systems to the market. Whether you focus on metrics, logging and tracing, or pipelines and visualization, you’ll have an outsize opportunity to enable both CoreWeave and its customers to understand, troubleshoot, and optimize complex systems at the forefront of Artificial Intelligence.

We are seeking senior observability engineers with specializations in logging and tracing platforms (where knowledge of Loki and/or ClickHouse is a strong plus), pipelines and visualisation (where knowledge of Kafka and Grafana will set you apart). As you work closely with a tightly knit group of observability experts, you will have the opportunity to:

• Build low-touch, automated tooling that enables CoreWeave engineers to clearly articulate which pieces of telemetry, visualizations, and dashboards should be presented to customers.
• Modernize logging platforms at cloud-scale.
• Design migrations that are transparent to platform consumers.
• Assist CoreWeavers in differentiating between optimal use cases for structured and unstructured logs.
• Build governance mechanisms that empower CoreWeavers to effectively manage the telemetry their services produce and adopt best practices.
• Assist engineers across CoreWeave’s in maximizing the insights they glean from all three pillars of observability.
• Develop and enforce best practices regarding the health of telemetry ETL pipelines.
• Improve the performance, security, reliability, and scalability of observability services while participating in the team’s on-call rotation.
• Work with telemetry data at enormous scale, ingesting data from industry-leading GPU clusters.
• Grow, change, invest in your teammates, be invested-in, share your ideas, listen to others, be curious, have fun, and, above all, be yourself.

Wondering if you’re a good fit? We believe in investing in our people, and value candidates who can bring their own diversified experiences to our teams – even if you aren't a 100% skill or experience match. Here are some qualities we’ve found compatible with our team. If a portion of this resonates with you, we’d love to talk. 

• You have six or more years of experience in a software or infrastructure engineering industry.
• You enjoy helping your colleagues achieve more with less effort.
• You are customer obsessed, ecstatic to provide infrastructure as a service, and default to adopting a product lens when evaluating platform scale problems.
• You have experience with logging and tracing platforms in production and at scale, and are versed in reliability engineering concepts such as the different types of testing, progressive deployments, error budgets, the role observability plays, and fault-tolerant design.
• You are opinionated about the best use cases for each of the three Observability pillars, and are excited about enabling engineers of all stripes - from frontend down to the kernel – with the best tools for their specific jobs.
• You have experience using Kubernetes, a conceptual understanding of its major components, and/or have operated Kubernetes clusters at scale for both event-driven and stateful orchestration.
• You’re familiar with various logging and metrics systems like ClickHouse, Elastic, Loki, Victoria Metrics, Prometheus, Thanos and Grafana. You have experience with designing and operating these systems at scale.
• You are familiar with PromQL, any other querying language and enjoy understanding the data model for observability systems. 
• You are excited at the prospect of zero-downtime, transparent migrations of major production services.