Senior Technical Product Manager, ML Quality

Who We Are: WellSaid 

WellSaid is the leading AI voice company for enterprise and professional applications. Using carefully sourced voice talent and our own AI advanced platform, WellSaid provides ultra-realistic voices that the world’s biggest brands trust to engage listeners. We build AI responsibly and ethically. 


Who You Are: An Experienced Technical Product Manager with a Focus on ML & Model Performance

You are an experienced product manager capable of managing the requirements, timelines, and overall roadmap of technical products in generative AI quality assurance. You have a strong technical background and a history of collaborating successfully in ML, TTS, or a related field. More specifically, in your previous role(s) you have been responsible for overseeing the testing and validation of ML models to guarantee their accuracy, reliability, and ethical implementation within a product.  Ideally, you have experience in audio and can manage technical conversations pertaining to audio signal qualities and methods for measuring and identifying them. 

You manage product delivery through the R&D flywheel: from research to development, to testing, to delivery, to feedback, and back to research. You write clear project plans and define requirements in a way that is comprehensive and can be handed off to engineers to design and develop against.  You rely on your research skills to inform your product requirements, test case suites, and testing plans. You work well with data engineers and analysts to make informed recommendations and evaluation reports. You are able to draw connections between model limitations and methods for measuring them.

Further, you take the ethics surrounding AI very seriously and work hard to protect all stakeholders in data gathering, research, and exposure. You are a creative thinker and strong collaborator who enjoys working with engineering teams to build effective solutions with a meaningful impact on our products. 

How You’ll Contribute:  

As Senior Technical Product Manager, ML Quality, you will manage the development of products and processes that are used to evaluate data, models, and features developed by the ML team. Your evaluations may result in reports highlighting major model developments to the broader WS community or in guidance for the development team honing various TTS features. For example, you might ensure the quality and accuracy of new TTS models before they launch to customers through data research, model testing, and regression analysis.

You will be responsible for:

• Defining QA strategy for ML models across many languages: Identifying critical metrics and testing methodologies to assess the performance of TTS models throughout the research & development lifecycle. This includes developing such strategies for any new languages we explore in the future.
• Evaluating and validating TTS models: Working with the Applied ML team and selecting appropriate evaluation metrics, running rigorous tests across datasets, experiments, and final models, and interpreting results to identify areas for improvement and/or offer recommendations to the Research team. 
• Managing data quality:  Monitoring data quality, identifying potential biases in training data, and ensuring data preprocessing is done correctly to optimize model performance. 
• Developing QA automation frameworks: Defining automated testing tools and pipelines to streamline the testing and evaluation processes for TTS models, especially for regression testing. 
• Monitoring model performance in production: Tracking key metrics post-deployment to identify potential degradation in model performance and take corrective action. This may also require defining the monitoring and tooling needed to do so.
• Moderating ethical considerations: Assessing potential biases in ML models and training data, ensuring responsible development practices, and addressing concerns regarding fairness, transparency, and Responsible AI. 
• Researching and partnering with third parties: Defining test cases, scenarios, and evaluation batches with external evaluation services when appropriate. Recommending evaluation services to leadership with a well-defined testing strategy defending your recommendation.

In your day-to-day, you will:  

• Collaborate closely with data engineers and software engineers in the ML team to understand model capabilities, identify new methods or metrics for gauging model progression, and create new tools to make the testing and evaluation processes fast and easy to implement
• Write design docs with clear requirements and expectations for engineers to e