Senior AI QA Engineer
TruistAbout the role
The position is described below. If you want to apply, click the Apply Now button at the top or bottom of this page. After you click Apply Now and complete your application, you'll be invited to create a profile, which will let you see your application status and any communications. If you already have a profile with us, you can log in to check status.
If you have a disability and need assistance with the application, you can request a reasonable accommodation. Send an email to Accessibility (accommodation requests only; other inquiries won't receive a response).
Regular or Temporary:
RegularLanguage Fluency: English (Required)
Work Shift:
1st shift (United States of America)Please review the following job description:
***This position is 5 days a week in the Charlotte Office***The Senior AI QA Engineer owns the quality validation of AI-enabled applications, agent workflows, and non-deterministic outputs in production-style enterprise environments.
This role combines the rigor of senior software quality engineering with the specialized evaluation discipline required for AI systems where correctness, completeness, hallucination risk, bias, safety, and reliability must all be measured explicitly.
This is a hands-on role focused on testing AI-powered products, agent-integrated workflows, prompt-driven solutions, and enterprise automations. The engineer designs and executes quality strategies spanning functional testing, regression testing, API validation, UI verification, output scoring, hallucination detection, evaluation dataset design, drift monitoring, and deployment gate readiness for agentic systems.
The role operates within the Forge model: production-first, governance-aware, and evidence-driven. Daily work includes defining test approaches for AI workflows, building automated test assets, validating prompts and outputs, running scenario-based and adversarial checks, partnering with engineering to remediate defects, and ensuring AI-enabled capabilities meet quality, security, and operational standards before release.
***For this opportunity, Truist will not sponsor an applicant for work visa status or employment authorization, nor will we offer any immigration-related support for this position (including, but not limited to H-1B, F-1 OPT, F-1 STEM OPT, F-1 CPT, J-1, TN-1 or TN-2, E-3, O-1, or future sponsorship for U.S. lawful permanent residence status.)***
ESSENTIAL DUTIES AND RESPONSIBILITIES
Following is a summary of the essential functions for this job. Other duties may be performed, both major and minor, which are not mentioned below. Specific activities may change from time to time.
- Design and execute quality strategies for AI-powered applications, agents, prompts, and workflow automations, covering functional behavior, output quality, regression risk, and production readiness.
- Validate AI outputs for accuracy, completeness, consistency, policy adherence, and business usefulness using defined scoring rubrics, golden datasets, and repeatable evaluation methods.
- Build and maintain manual and automated test assets for UI, API, workflow, and agent-integrated solutions, including tests for prompts, tools, memory behavior, and non-deterministic outputs.
- Conduct hallucination, drift, and quality degradation testing for agentic systems and AI-enabled workflows before and after model, prompt, or data changes.
- Partner with engineering teams to investigate failures, isolate root causes, and drive remediation for defects affecting AI output quality, system behavior, or release confidence.
- Define and enforce quality gates for AI features and deployments, including acceptance thresholds for output quality, test coverage, observability, and release readiness.
- Support evaluation of tool-calling behavior, workflow branching, human-in-the-loop steps, and failure handling across complex AI or agentic user journeys.
- Create and
Apply for this role
Generate a tailored application kit with a matched cover letter, interview prep, and CV highlights — in under 60 seconds.
Apply Now →Generate Application KitFree account required — sign up in 30s