Senior Applied AI Engineer (Observability)

We Breathe Life Into Data

At Komodo Health, our mission is to reduce the global burden of disease. And we believe that smarter use of data is essential to this mission. That’s why we built the Healthcare Map — the industry’s largest, most complete, precise view of the U.S. healthcare system — by combining de-identified, real-world patient data with innovative algorithms and decades of clinical experience. The Healthcare Map serves as our foundation for a powerful suite of software applications, helping us answer healthcare’s most complex questions for our partners. Across the healthcare ecosystem, we’re helping our clients unlock critical insights to track detailed patient behaviors and treatment patterns, identify gaps in care, address unmet patient needs, and reduce the global burden of disease. 

As we pursue these goals, it remains essential to us that we stay grounded in our values: be awesome, seek growth, deliver “wow,” and enjoy the ride. At Komodo, you will be joining a team of ambitious, supportive Dragons with diverse backgrounds but a shared passion to deliver on our mission to reduce the burden of disease — and enjoy the journey along the way.

The Opportunity at Komodo Health: 

Healthcare in the U.S. is fragmented and inefficient. Komodo Health is working to fix that—with data.

By building the most complete view of patient journeys across the U.S., Komodo enables life sciences companies, payers, and providers to make better decisions that directly improve patient outcomes.To unlock the full potential of this data, Komodo is investing in AI-native infrastructure—systems that make AI reliable, scalable, and deeply embedded into how products are built and used.

Mission of the Role:

We’re hiring Senior Applied AI Engineers to join a newly formed AI Platform / Observability team focused on building the foundation that makes AI systems trustworthy at scale. This is a backend-heavy, systems-focused role in a greenfield environment. You’ll own problems end-to-end—from early exploration through production deployment—while helping define how AI systems are built, evaluated, and operated across Komodo.

Your starting point will be AI observability, evaluation, and production reliability. From there, your scope expands into broader platform ownership: agent systems, orchestration layers, and shared infrastructure.

Observability is the entry point—not the end state.

Looking back on your first 12 months at Komodo Health, you will have accomplished… 

• Build the observability and reliability foundation for AI systems across Komodo (logging, tracing, evaluation pipelines, feedback loops)
• Define how the organization measures LLM performance and quality in production (hallucinations, drift, latency, failure modes)
• Ship production-grade AI systems that improve platform reliability, scalability, and performance
• Lead design and architecture for complex applied AI systems (multi-agent workflows, tool-calling systems, model pipelines)
• Establish evaluation frameworks and experimentation practices (A/B testing, offline + online evaluation)
• Contribute to reusable infrastructure, patterns, and standards adopted across teams

What You'll Own:

AI Observability & Reliability (Initial Focus)

• Design and implement:
  • Logging, tracing, and request visibility for LLM systems
  • Evaluation pipelines and benchmarking frameworks
  • Feedback loops for continuous system improvement
• Define metrics for:
  • Output quality and correctness
  • Latency and system performance
  • Tool usage and agent behavior
• Detect and debug: hallucinations, model drift and system degradation and failure modes

Applied AI Systems & Platform (Expanded Scope)

• • Architect and deploy end-to-end AI systems agent-based workflows, prompt chains and tool integrations and scalable LLM-powered services
  • Transition prototypes into reliable, production-grade systems
  • Contribute to shared AI infrastructure and orchestration patterns
  • Partner with product, data, and platform teams to shape AI-driven solutions

What you bring to Komodo Health (required):

• Experience building production AI systems end-to-end (not just prototypes)
• Strong expertise with LLMs and prompt systems and agent orchestration and tool/function calling
• Hands-on experience with:
  • AI observab