Software QA Lead – AI & GenAI Platforms

Company Description

Work with Us. Change the World.

At AECOM, we're delivering a better world. Whether improving your commute, keeping the lights on, providing access to clean water, or transforming skylines, our work helps people and communities thrive. We are the world's trusted infrastructure consulting firm, partnering with clients to solve the world’s most complex challenges and build legacies for future generations.

There has never been a better time to be at AECOM. With accelerating infrastructure investment worldwide, our services are in great demand. We invite you to bring your bold ideas and big dreams and become part of a global team of over 50,000 planners, designers, engineers, scientists, digital innovators, program and construction managers and other professionals delivering projects that create a positive and tangible impact around the world.

We're one global team driven by our common purpose to deliver a better world. Join us.

Job Description

Build the quality engine behind enterprise AI.

We’re looking for a hands-on QA leader to drive the strategy and execution of quality assurance for enterprise-grade Generative AI assistant applications.

You’ll own how AI systems are tested, validated, and scaled, spanning full-stack applications powered by LLMs and RAG workflows. This role is critical to ensuring reliable AI outputs, strong user experiences, and responsible, ethical AI behavior in production.

This position will offer flexibility for hybrid work schedules to include both in-office presence and telecommute/virtual work, to be based in Chicago, IL.

Key Responsibilities

• Define and execute QA strategy for end-to-end AI applications (frontend, backend, and AI workflows)

• Lead validation of LLM outputs for accuracy, coherence, and reliability across diverse scenarios

• Test and optimize RAG pipelines to ensure high-quality retrieval and grounded responses

• Design user-centric test scenarios (multi-turn conversations, edge cases, failure handling)

• Conduct performance, load, and latency testing across AI and application layers

• Build and scale automated testing frameworks (API, UI, integration) within CI/CD pipelines

• Establish and track quality metrics (accuracy, relevance, latency, user satisfaction)

• Ensure ethical AI standards, including bias detection and responsible output validation

• Partner cross-functionally to embed QA practices across the development lifecycle

Qualifications

Minimum Qualifications

• Bachelor's Degree plus at least 10 years of software development experience, or demonstrated equivalency of experience and/or education

• Proven experience leading QA for complex, full-stack applications

• Strong expertise in testing Generative AI systems (LLMs, RAG workflows)

• Hands-on experience with test automation tools (e.g., Selenium, Appium)

• Experience with API testing, distributed systems, and CI/CD pipelines

• Familiarity with performance testing tools (e.g., JMeter, LoadRunner)

• Strong analytical, problem-solving, and communication skills

Preferred Qualifications

• Experience with AI evaluation techniques (prompt testing, output scoring, bias testing)

• Background in conversational AI, NLP, or chatbot testing

• Familiarity with enterprise data/retrieval systems and knowledge bases

• Experience working in cloud environments (AWS, Azure, or GCP)

Additional Information

Additional Information

• Relocation assistance is not available for this position

• Sponsorship for US work authorization is not available for this position, now or in the future.

About AECOM 

AECOM is proud to offer comprehensive benefits to meet the diverse needs of our employees. Depending on your employment status, AECOM benefits may include medical, dental, vision, life, AD&D, disability benefits, paid time off, leaves of absences, voluntary benefits, perks, flexible work options, well-being resources, employee assistance program, business travel insurance, service recognition awards, retirement savings plan, and employee stock purchase plan. 

AECOM is the global infrastructure leader, committed to delivering a better world. As a trusted professional services firm powered by deep technical abilities, we solve our clients’ complex challenges in water, environment, energy, transportation and buildings. Our teams partner with public- and private-sector clients to create innovative, sustainable and resilient solutions throughout the projec