Lead Quality Engineer - AI/ML

As a leading industrial distributor with operations primarily in North America, Japan and the United Kingdom, We Keep The World Working® by serving more than 4.5 million customers worldwide with products delivered through innovative technology and deep customer relationships. With 2023 sales of $16.5 billion, we’re dedicated to providing value for customers, fostering an engaging culture for team members and driving strong financial results.

 

Our welcoming workplace enables you to learn, grow and make a difference by keeping businesses running and their people safe. As a 2024 Glassdoor Best Place to Work and a Great Place to Work-Certified™ company, we’re looking for passionate people to join our team as we continue leading the industry over our next 100 years.

 

 

Position Details:

A new and rapidly growing team at Grainger is focusing on transforming a variety of transactional and operational data, to support the development of new tools and services aimed at enhancing our ability to respond to Customer inquiries. 

 

We are looking for a  LLM Testing Engineer specializing in automated testing frameworks for language models (LLMs). You will develop and execute automated scripts to evaluate the accuracy, reliability, and efficiency of our LLM outputs. You will collaborate closely with our AI/ML, QA, and engineering teams to build robust testing frameworks, improve model reliability, and ensure seamless, consistent user interactions.

 

You will report to the Product Engineering Manager

You Will:

• Define test automation strategy and lead implementation of automated solutions to test LLM responses across multiple scenarios, including edge cases and diverse prompt inputs.
• Analyze and assess the language model's outputs for quality, accuracy, coherence, and adherence to requirements.
• Build and maintain testing frameworks and tools to automate and streamline the testing process, ensuring scalable, efficient testing for large language models.
• Use statistical tools to analyze test results, identify patterns in model errors, and generate reports for the ML and engineering teams.
• Perform manual testing on LLM outputs to address urgent needs, high-stakes scenarios, or particularly challenging prompt responses to ensure quality.
• Conduct regular and on-demand regression testing, unit and integration tests ensuring the model’s performance remains stable after updates, particularly for high-impact changes.
• Collaborate with engineering teams to ensure the LLM meets high standards for deployment, establishing benchmarks for response quality, performance and consistency.
• Maintain and update test cases, procedures, and detailed documentation for automated and manual test scripts. Provide clear documentation for reproducibility and knowledge-sharing.
• Design and implement test strategies and scripts targeting API, and front-end UI, with potential expansion to other components as needed.
• Identify, document, and report issues in model outputs, ensuring quick turnaround on urgent fixes in collaboration with engineering and QA.
• Testing in a Cloud/AWS, CI/CD DevOps environment.
• Collaborate closely with AI/ML engineers to provide actionable feedback on model behavior, issues and response generation.
• Work closely with AI/ML engineers, data engineers, and product managers to refine prompt strategies and model performance.
• Focus on increasing depth of understanding of domain and core systems maintained by team
• Lead production issue triage efforts

You Have:

• Bachelor's degree in Computer Science, Data Analysis/Science, Engineering, or a related field, or equivalent work experience.
• 6+ years in QA engineering, test automation, or LLM/AI model testing.
• Proficiency in scripting languages (e.g., Python, Java) and familiarity with test automation frameworks (e.g., Selenium, Robot Framework, Pytest).
• Familiarity with NLP models, large language models (LLMs), and prompt e