Principal Engineer - AI

Costco IT is responsible for the technical future of Costco Wholesale, the third largest retailer in the world with wholesale operations in fourteen countries. Despite our size and explosive international expansion, we continue to provide a family, employee centric atmosphere in which our employees thrive and succeed.

This is an environment unlike anything in the high-tech world and the secret of Costco’s success is its culture. The value Costco puts on its employees is well documented in articles from a variety of publishers including Bloomberg and Forbes. Our employees and our members come FIRST.  Costco is well known for its generosity and community service and has won many awards for its philanthropy. The company joins with its employees to take an active role in volunteering by sponsoring many opportunities to help others. 

Come join the Costco Wholesale IT family. Costco IT is a dynamic, fast-paced environment, working through exciting transformation efforts. We are building the next generation retail environment where you will be surrounded by dedicated and highly professional employees.

The Principal AI Engineer is the lead Architect and hands-on builder of our unified AI Platform as a Service. This role is responsible for transforming raw foundation model capabilities into a scalable, multi-tenant reasoning stack that empowers the entire enterprise to build, deploy, and manage semantic discovery, conversational intelligence, and autonomous agents. This role will balance 40% hands-on systems development with 60% platform strategy, personally coding the core orchestration engines, standardized capability servers, and universal trust guardrails. The mission is to provide a central ‘AI Operating System' for the company, ensuring that specialized agents across different business units can communicate via inter-agent protocols, access grounded knowledge layers securely, and execute autonomous tasks within a governed, high-performance agent runtime environment.

If you want to be a part of one of the worldwide BEST companies “to work for”, simply apply and let your career be reimagined.

ROLE

• Develops and implements an industry-leading, self-service AI platform. This platform will offer standardized blueprints for engineering teams to utilize, including "macro-agents" and "micro-tools."

• Develops and articulates the long-term, multi-year technical roadmap for the AI Platform, ensuring its capabilities are strategically aligned with the overarching business objectives.

• Develops and implements complex stategraphs to manage edge cases and enable self-correction in autonomous planning.

• Leads the architecture and hands-on development of remote MCP servers and implements custom function calling to securely connect agents with sensitive enterprise data.

• Defines and implements the communication standards for agent-to-agent interactions to facilitate autonomous discovery and task hand-offs between agents developed by various business units.

• Ensures the agent identity layer is architected for granular permissioning and non-repudiation, specifically regarding every autonomous system action.

• Develops a unified knowledge layer for the platform, leveraging semantic retrieval engine and multimodal grounding. This layer will serve as the single source of truth for all connected agents, providing "truth-as-a-service."

• Develops and implements a global memory bank architecture, leveraging semantic retrieval engine and graph databases. This system will be essential for preserving context and capturing "institutional knowledge" from interactions over time.

• Develops the platform's trust layer by automating rapid evaluation pipelines. These pipelines will measure key metrics of success, cost, and safety for agentic behavior across all tenants.

• Ensures the agent runtime environment lifecycle is managed to guarantee high availability, session persistence, and global scalability for the company's digital workforce.

• Performs in-depth, rigorous code reviews with a specific focus on identifying and mitigating the unique failure modes inherent in agentic systems, such as s