Executive Director, Natural Language Measurement

College Board – Learning and Assessment – Psychometrics  

100% Remote (anticipated 25% travel, usually 2-3 days at a time) 

About the Team

The Psychometrics Team supports College Board in serving more than seven million students a year as they prepare for a successful transition to college through programs and services in college readiness and college success. We provide our partners with measurement solutions that set the standard for quality, efficiency, and innovation through operational excellence and research leadership that advances the field of psychometrics and supports College Board’s mission of access and equity in education. Our team of Psychometricians and Research Statisticians collaborates deeply with other departments and divisions including, but not limited to Assessment Design & Development, College Readiness Assessments, Advanced Placement & Instruction, State & District Partnerships, and Technology. 

About the Opportunity  

As our Executive Director, Natural Language Measurement, you are a strategic measurement leader at the forefront of responsible AI in education. Energized by our mission to enhance educational and career opportunities for millions of students, you will shape the future of automated scoring and use of AI-powered feedback in large-scale assessments. You will set the vision and multi-year roadmap for natural language measurement across SAT, AP, CLEP, Accuplacer and future products, ensuring all applications are grounded in validity, fairness, and reliability. You will expand the College Board’s innovation agenda in AI-enabled approaches that power personalized learning and feedback, reinforcing our leadership in responsible and equitable assessment. You bring experience shaping national or international best practices in responsible AI/NL for large-scale assessment and have a proven record of scaling applications from research to operational use in high-stakes contexts, making a transformative impact on both students and the field.

In this role, you will:

Strategic Leadership and Roadmap (35%)

• Shape and implement the multi-year strategy and roadmap for automated scoring and intelligent learning approaches across current flagship programs, such as SAT and AP, and emerging future products.

• Advance College Board’s innovation agenda by expanding from automated scoring to broader AI-enabled solutions that support personalized learning and actionable feedback.

• Establish model-readiness criteria and guide release decisions for senior leadership to ensure high-quality, scalable deployment of natural language-driven solutions.

• Partner closely with Technology to shape development of guided scoring products and other intelligent learning innovations, advancing next-generation assessment capabilities.

Operational and Implementation Work (35%)

• Lead scientific design, implementation, and evaluation of automated scoring and AI/NL applications, ensuring alignment with program and organizational goals.

• Apply psychometric and measurement principles to ensure all AI/NL-driven applications uphold validity, fairness, reliability, and equity.

• Guide implementation of initial automated scoring programs beginning in 2026.

• Establish and oversee operational processes that ensure quality, fairness, and reliability, including monitoring, human-in-the-loop procedures, and refinement of evaluation criteria.

• Build organizational readiness for adoption of new technologies, including criteria for model evaluation, quality assurance and responsible release decisions.

Lead and Manage the Natural Language Measurement Team (20%)

• Build and lead a team of psychometricians, language model experts, and related specialists, combining internal talent with new hires to deliver AI/NL priorities.

• Set vision and priorities for the team, track and manage progress to goals, and provide coaching and support to ensure team members meet and exceed goals, remain engaged, and contribute meaningfully to our mission.

• Cultivate an inclusive and high-achieving culture that enables all team members to live out College Board’s Operating Principles effectively.

• Bring proven leadership experience and consistently embody College Board’s Manager Expectations in your work.

Internal and External Collaboration and Communication (10%)