AI Quality Assurance Specialist Sr. Lead

Job Title: AI Quality Assurance Specialist Sr. Lead

Work Place Flexibility: Hybrid 

Legal Entity: Entergy Services, LLC 

***The preferred location for this role is New Orleans, LA. Other locations within Entergy’s service territory may be considered***

 

JOB SUMMARY/PURPOSE

This role blends traditional software QA skills with advanced evaluation methodologies for modern AI models, including LLMs and ML-driven product features. As a member of the team, you will work closely with experienced engineers and machine learning experts to qualify and refine features powered by vision, language, and cross-modal models. You will be responsible for designing rigorous evaluation strategies for both objective and subjective ML behaviors, creating reliable automated testing pipelines, and developing LLM-driven evaluators that complement human judgement. The ideal candidate is self-directed, creative, and comfortable with ambiguity, with strong technical and interpersonal skills. They have hands-on experience testing ML models directly, defining qualitative scoring rubrics, building reproducible evaluation frameworks, and ensuring that AI behavior is safe, consistent, and aligned with product vision.

 

JOB DUTIES/RESPONSIBILITIES

· Create detailed, comprehensive, and well-structured test plans and test cases for traditional and ML-based solutions. Estimate, prioritize, plan, and coordinate testing activities.

· Design, develop, and execute automation scripts. Identify, record, document, and track bugs. Perform thorough regression testing when bugs are resolved.

· Develop and maintain robust testing frameworks for machine learning models

· Develop and apply testing processes for new and existing products to meet client needs.

· Monitor debugging process results. Investigate the causes of non-conforming software. Track quality assurance metrics, such as defect densities and open defect counts.

· Collaborate with internal teams (e.g. developers, product managers, and designers) to identify testing requirements.

· Identify and prioritize quality control issues from an end-to-end perspective.

· Participate in scrum ceremonies such as refinement, demo, and release planning.

 

MINIMUM REQUIREMENTS

Minimum education required of the position: Bachelor’s degree in computer science, MIS, business administration, or other technical area of study OR equivalent work experience; advanced degree is a plus.

Minimum experience required of the position:

Specialist I: 0-3 years of professional experience. Specialist II: At least 2 years of professional experience, at least 1 year of experience testing ML/GenAI solutions. Specialist III: At Least 4 years of professional experience, at least 2 years of experience testing ML/GenAI solutions. Specialist, Sr: At least 6 years of professional experience, at least 3 years of experience testing ML/GenAI solutions. Consultant: At least 8 years of professional experience, at least 4 years of experience testing ML/GenAI solutions. Consultant, Sr: At least 10 years of professional experience, at least 5 years of experience testing ML/GenAI solutions.

 

Minimum knowledge, skills and abilities required of the position:

· Strong knowledge of software QA methodologies, tools, and processes.

· Familiarity working with Agile development teams.

· Good communications skills; ability to effectively communicate verbally and in writing with both non-technical and technical staff.

· Proficient in both white box and black box testing methods.

· Ability to work both independently and as part of a team.

 

Desired knowledge, skills, and abilities:

· Working knowledge of automated testing tools.

· Highly proficient SQL and scripting skills.

· Experience testing AI/ML systems in cloud-based environments.

· Experience integrating automated AI testing into CI/CD pipelines (e.g., GitHub Actions or similar tools).

· Experience with monitoring and logging systems for post-deployment model validation.

· Advanced experience testing generative AI systems, including LLMs for accuracy, bias, safety, a