Clinical AI Engineer

We are seeking a Clinical AI Engineer to join our innovative team working at the intersection of healthcare and AI. This role focuses on designing, evaluating, and improving LLM-powered tools that support abstraction and data analysis for clinical registries. The ideal candidate combines clinical domain expertise with a deep understanding of AI prompt engineering, model evaluation, and data quality analysis to build scalable solutions for real-world healthcare workflows.

As a Clinical AI Engineer, you’ll work closely with product, engineering, and clinical operations teams to define, test, and refine AI behaviors across a portfolio of registry automation tools. You’ll design prompts, evaluate AI outputs, and build systems that ensure safe, accurate, and performant abstraction across diverse patient populations and registry types.

Responsibilities:

Design and Evaluation of LLM Systems

• Develop and refine prompt strategies tailored to clinical registry questions and workflows.
• Design evaluation protocols and behavioral test suites to assess the performance of LLM-based systems for accuracy, relevance, and clinical appropriateness.
• Partner with ML/engineering teams to test novel prompting techniques, chain-of-thought reasoning, and retrieval augmentation in abstraction workflows.

Clinical and Registry Expertise

• Translate complex registry data requirements into structured abstraction workflows using LLMs.
• Apply clinical reasoning and judgement to identify subtle errors, gaps, and contextual misunderstandings in AI-generated outputs.
• Leverage experience with clinical data, abstraction processes, and EHR documentation to improve real-world usability and accuracy.

Data Quality, Analysis, and Measurement

• Design metrics and dashboards to measure model coverage, abstraction precision/recall, and user correction rates.
• Build error taxonomies and track failure modes across registry products to drive continuous improvement.
• Collaborate with engineering and research teams to recommend architectural or training enhancements to LLM systems.
  
  

Product and Team Collaboration

• Contribute clinical insight and feedback to roadmap development for registry abstraction and data quality tooling.
• Work closely with nurses, abstractors, and clinical informaticists to validate abstraction behavior and usability.
• Pair with software engineers and AI researchers to test product hypotheses and bring features to production.

What you’ll need:

Clinical Knowledge & Experience

• Strong foundation in interpreting patient data including diagnoses, labs, procedures, and treatment timelines.
• Experience working directly with registry programs (e.g., STS, NSQIP, ACC, etc.) and abstracting clinical data.
• Familiarity with nursing workflows and how clinicians interact with EHRs and abstraction interfaces.

AI Prompt Engineering and Evaluation

• 1+ year hands-on experience designing prompts for LLMs in a professional or research setting.
• Strong intuition for how AI systems interpret instructions and when/how they fail in clinical contexts.

Experience designing test sets and evaluation pipelines for AI output behavior.

Data QA and Measurement

• Experience measuring performance of AI or human-in-the-loop systems with precision, recall, and coverage metrics.
• Familiarity with clinical data QA principles and constructing reproducible test environments.

Ability to create and use error taxonomies to drive iterative improvement of AI behaviors.

Additional Skills

• Excellent written and verbal communication with both technical and clinical stakeholders.
• Comfortable working in fast-paced, cross-functional teams where responsibilities may span research, engineering, and operations.

Bachelor’s degree in a clinical field (e.g., Nursing, Health Informatics, Public Health) or equivalent experience. Master’s degree or relevant AI coursework is a plus.

Why we love Carta Healthcare, and why you will too!

• Working on a new, challenging problem space with LLMs/AI/Healthcare
• Industry leading products
• Work hard, and have fun doing it
• Work alongside some of the most talented and dedicated teammates
• Mission driven 
• Competitive benefits package including great healthcare benefits and 401k

 

All applica