Senior Instrumentation & Observability Engineer

Description

Position Summary:

The Senior Instrumentation and Observability Engineer leads the design, implementation, and maintenance of advanced observability strategies for complex distributed systems. This role is responsible for architecting sophisticated monitoring solutions, establishing observability standards, and driving the adoption of best practices across the organization. As a senior technical contributor, you will provide deep expertise in telemetry data collection, analysis, and visualization while mentoring team members and influencing observability practices throughout the engineering organization.

Essential Functions and Job Responsibilities:

Observability Architecture & Strategy

• Architect comprehensive observability solutions for complex distributed systems and microservice architectures
• Define the long-term technical vision and strategy for observability across the organization.
• Establish best practices, standards, and patterns for instrumenting applications and infrastructure.
• Lead cross-team initiatives to improve system observability and reduce mean time to detect/resolve issues.
• Design scalable telemetry data collection, processing, and storage systems capable of handling high volumes.

Advanced Monitoring & Analysis

• Design sophisticated monitoring systems with advanced alerting logic and minimal alert fatigue.
• Develop comprehensive SLI/SLO frameworks aligned with business objectives.
• Create advanced visualization systems providing actionable insights into system behavior and performance.
• Implement anomaly detection and predictive analytics to identify potential issues before they impact users.
• Lead post-incident reviews with data-driven analysis to prevent recurrence of issues.

Leadership & Mentorship

• Serve as a technical leader and subject matter expert on observability across the organization.
• Mentor junior engineers and provide guidance on observability practices and techniques.
• Collaborate with engineering leadership to define and implement observability roadmaps.
• Lead the evaluation and adoption of new observability technologies and approaches.
• Partner with product and development teams to integrate observability considerations into the design phase.

Platform Innovation

• Architect and develop custom observability solutions for unique business and technical requirements.
• Lead the design and implementation of observability data pipelines with sophisticated processing capabilities.
• Create advanced self-service tools that empower teams to manage their observability configurations.
• Develop integrations between observability systems and other enterprise platforms.
• Optimize observability infrastructure for cost-effectiveness and efficiency.
• Maintain patient confidentiality and function within the guidelines of HIPAA.
• Complete assigned compliance training and other educational programs as required.
• Maintain compliance with AdaptHealth’s Compliance Program.
• Perform other related duties as assigned.
• Assist in vendor contract reviews with managers and legal.

Competency, Skills, and Abilities:

• Exceptional problem-solving abilities with systematic approach to complex issues
• Outstanding communication skills with ability to explain technical concepts to varied audiences.
• Strategic thinking with ability to balance immediate needs and long-term vision.
• Demonstrated leadership with ability to influence without direct authority.
• Proactive mindset with focus on continuous improvement
• Ability to mentor others and share knowledge effectively.
• Experience building and leading observability teams or functions.
• Expertise in multiple cloud platforms and their native observability solutions
• Contributions to open-source observability projects or published work on observability topics.
• Experience implementing observability in high-compliance environments (finance, healthcare)
• Background in statistical analysis, data science, or machine learning as applied to system monitoring.
• Experience with AIOps and automated remediation systems
• Observability Platforms: Expert-level knowledge of multiple platforms (Prometheus, Grafana, Datadog, New Relic, Elastic Stack, Splunk)
• Distributed Tracing: Deep experience with OpenTelemetry, Jaeger, Zipkin and trace-based analysis
• Programming: Advanced proficiency in Python, Go, Java, or similar languages
• Infrastructure: Expert knowledge of Kubernetes, service mesh, and cloud platforms (AWS, GCP, Azure)
• Data Engineering: Advanced experience with time-series databases,