Senior Product Manager

Summary:

The Senior Product Manager will join the market-leading team at Thomson Reuters Westlaw and be at the forefront of transforming legal research and litigation workflows with generative AI. We are seeking a strategic and technically adept Senior Product Manager to spearhead the development and implementation of automated evaluation systems for our cutting-edge AI models. This is a unique opportunity to leverage your deep legal understanding and product expertise to ensure the accuracy, reliability, and trustworthiness of AI-generated content used by legal professionals worldwide. You will work closely with world-class product managers, research scientists, and engineers, playing a critical role in shaping the future of legal technology.

About the Role:

• Strong Legal Research: Break down complex legal research and related litigation workflows to identify how generative AI can enhance specific tasks (legal research, brief analysis, motion improvement) and define the evaluation criteria necessary to ensure AI outputs meet the high standards of legal practice.

• Evaluation Strategy & Framework Development: Define, design, and drive the implementation of robust, scalable automated evaluation frameworks and metrics for generative AI in the legal domain, aligning with business goals and product roadmaps. Assess critical factors like factual accuracy, legal soundness, citation validity, hallucination detection, bias mitigation, and relevance. 

• Data Management, Testing & Validation: Oversee the creation and curation of high-quality datasets for auto-evaluation, including both ground truth data and adversarial examples. Lead the execution of automated testing and validation pipelines, identifying and addressing performance bottlenecks and areas for improvement.

• Cross-Functional Leadership: Partner effectively with research scientists, engineers, attorney editors, and product managers to translate complex requirements into actionable plans, align on evaluation priorities, and ensure seamless execution of generative AI initiatives.  

• Technical Insight & Innovation: Stay abreast of the latest advancements in LLMs, generative AI evaluation techniques, and prompt engineering. Apply this knowledge to innovate and continuously improve our evaluation methodologies and tools.

• Stakeholder Communication: Articulate evaluation strategies, results, risks, and insights to diverse audiences, including senior leadership, technical teams, and legal experts, driving informed decision-making.

• Team Leadership: Mentor and guide junior members of the auto evaluation team, fostering a culture of excellence and innovation.

• Market & User Insights: Conduct market research, competitive analysis, and user research (interviews, surveys, user testing) to deeply understand the needs, pain points, and expectations of legal professionals regarding AI-generated content and its evaluation.

About You:

You are a fit for the role of Senior Product Manager if your background includes:

• Legal Expertise (Required): J.D. from an accredited law school and a deep understanding of legal research methods, concepts, and their application within litigation workflows. Ability to dissect complex legal issues.

• Product Management Fundamentals (Required): 5+ years managing the end-to-end lifecycle (concept to launch) of complex technical products/features. Ability to lead customer research, specifically with experienced litigators, to understand workflows and translate needs into product requirements, especially for evaluation.

• AI & Technical Proficiency (Required): Minimum 1 year working with generative AI; fluency with current tools/techniques (Legal AI focus highly preferred). Experience developing/implementing auto-evaluation frameworks for GenAI models; strong grasp of relevant metrics/techniques for NLP. High-level familiarity with Large Language Model architecture and operation. Understanding of data management and quality best practices for AI. Experience working directly with data science teams on model processes (training, tuning, evaluation) and comfort with intermediate programming, specifically Python, preferred.

• Analytical & Strategic Problem-Solving: Excellent critical thinking, strategic analysis, and creative problem-solving abilities. Ability to clarify unclear problems and identify core, transferable principles. Proactively anticipate risks, model scenarios, and provide actionable guidance. Creative approach to leveraging technology and assessing new tech risks/benefits beyond stand