Principal Data Scientist

At Microsoft, our mission is to empower every individual and organization on the planet to achieve more, and we recognize the transformative power of artificial intelligence in realizing this vision. The Bing Metrics and Evaluation team is at the forefront of this journey, meticulously measuring and enhancing the user experience across Bing and Copilot. Our goal is to execute this with precision and efficiency.

This role is available in Mountain View, CA, Boulder, CO or Redmond, WA.

In this role as a Principal Data Scientist, you will work to improve online metrics that evaluate Product quality in A/B testing and non-A/B testing situations. You will apply sophisticated statistical methods and large language models (LLMs) combined with human labels to conduct thorough evaluations of Bing’s Search Product through the lens of metrics. Your expertise will guide Bing’s product quality assessments, and user satisfaction measurement.

As part of the rapidly advancing Search and AI domain, our team is a leader in applied machine learning at Microsoft. We are committed to providing an unparalleled search and AI assistant experience to over 500 million monthly active users globally through Bing. Additionally, our work impacts other major search engines like Yahoo, DuckDuckGo, and emerging platforms such as Neeva and You.com.

Join us in shaping the future of search and AI, where your contributions will influence the experiences of users worldwide.

Responsibilities

General Product quality measurement: Define, invent, and deliver metrics that accurately measure the quality of online information and the satisfaction/success of our customers.

A/B Metrics: Design and implement A/B metrics to evaluate the impact of changes to our products and services. Analyze results to provide actionable insights and drive continuous improvement.

RAG based search experience measurement: Develop and refine metrics to assess the performance and effectiveness of our Generative SERP experience. Ensure these metrics capture user satisfaction, accuracy, and overall interaction quality.

Models: Develop ML/Statistical models to measure/predict the quality of online content and/or user interactions with large-scale AI systems.
Experimental Design: Think critically about experimental design & develop innovative strategies and products in these areas to improve the quality of A/B experimentation

Strategy: Translate strategy into plans that are clear and measurable, with progress shared out to stakeholders.

Cooperation: Partner effectively with program management, engineers, and other areas of the business.

Qualifications

Required Qualifications:

• Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 5+ year(s) data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
  • OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 7+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) 
  • OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 10+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) 
  • OR equivalent experience. 

• 5+ years experience programming experience utilizing Python, R or Big Data environments

Preferred Qualifications:

• Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 8+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) 
  • OR Master's Degre