Data Scientist, GenAI Studio

College Board – Technology Division – GenAI Studio 
Location: This is a fully remote role. Candidates who live near CB offices have the option of being fully remote or hybrid (Tuesday and Wednesday in office).  
Type: This is a full-time position 

About the Team  
The GenAI Studio is an internal incubator at College Board that launched in January 2024. Through the Studio, College Board is investing time and resources for Generative AI ideation. The Studio is responsibly experimenting with AI internally with staff, in the development of our products, and even in AP classrooms. Our rapid prototyping and experimentation help us make informed decisions about AI in our broader portfolio of offerings. The Studio is comprised of a diverse, cross-functional team of strategists, technologists, and product managers at College Board committed to solving the most impactful and interesting problems arising with AI. Across the team we divide the work around strategic enablement within our organization, transformative AI innovations that are both internally and externally facing and advancing AI Literacy in the education space. 

About the Opportunity   
As a Data Scientist for our GenAI Studio, you are passionate about applying new ideas and technologies to creatively solve challenging, real-world problems. You are an early-adopter who is enthusiastic about finding high-value applications for emerging tools and new technology. You are highly capable at evaluating large data sets and telling a story with data. You are versatile, adaptable, and comfortable with uncertainty and ambiguity. You are resourceful and able to forge ahead even when faced with incomplete information. While representing our Data Science functional team, you will work day-to-day within our Strategy team where you will experiment with different generative AI techniques and tools to find the best solution for the students and teachers who use our services. You will work on cross-functional projects that involve data analysis, machine learning, natural language processing, large language models, model evaluation, and more. You will also help us define and build infrastructure to support rapid prototyping and testing of AI solutions. You will have the opportunity to learn from and collaborate with other data scientists, strategists, analysts, and project managers who share your passion for innovation and impact.  We recognize that GenAI is an evolving field. If you're excited about this space and meet most (but not all) of these qualifications, we encourage you to apply. 

In this role, you will:  

Prototype new tools leveraging emerging technologies and models (50%) 

• Collaborate with users, product and strategy teams to scope and define problem statements 
• Guide product iteration using model results, user feedback, and experimentation insights 
• Use orchestration and prompt engineering techniques to build exploratory solutions (e.g., LangChain, LlamaIndex) 
• Independently manage and complete all the work needed to deliver in a rapid-prototyping environment 
• Rapidly prototype user-facing AI tools to test and validate hypotheses 
• Create customized or multi-staged models to address nuanced use cases 
• Analyze structured and unstructured data to uncover patterns and inform model usage 

Evaluate generative AI models and tools (30%) 

• Utilize state-of-the-art techniques to evaluate generative model outputs—including off-the-shelf, open-source, and custom models—using quantitative and qualitative methods 
• Judiciously evaluate the feasibility, fairness, and effectiveness of GenAI solutions and articulate considerations and concerns around implementing models in the context of specific business applications 
• Translate complex findings into clear insights for technical and non-technical audiences 

Learning, ideating, and sharing (20%) 

• Keep abreast of rapid technological developments and relevant product offerings, especially within the Generative AI space 
• Workshop new ideas with the immediate team and program stakeholders 
• Share knowledge of new developments with the team via chats, recurring meetings, and presentations when appropriate 
• Communicate clearly with non-technical stakeholders to enhance their understanding of what the technology is and is not capable of 

About you, you have: 

• 5+ years' experience in data science, product analytics, or adjacent fields with a focus on applied machine learning 
• Strong analytical and product thinking skills with a track record of delivering real-world impact 
• Experience working with large language models, generative AI APIs, or orchestration frameworks 
• Hands-on proficiency with Pytho