Principal Machine Learning Engineer (AI Agents)

Company Description

Our Mission

At Palo Alto Networks® everything starts and ends with our mission:

Being the cybersecurity partner of choice, protecting our digital way of life.
Our vision is a world where each day is safer and more secure than the one before. We are a company built on the foundation of challenging and disrupting the way things are done, and we’re looking for innovators who are as committed to shaping the future of cybersecurity as we are.

Who We Are

We take our mission of protecting the digital way of life seriously. We are relentless in protecting our customers and we believe that the unique ideas of every member of our team contributes to our collective success. Our values were crowdsourced by employees and are brought to life through each of us everyday - from disruptive innovation and collaboration, to execution. From showing up for each other with integrity to creating an environment where we all feel included.

As a member of our team, you will be shaping the future of cybersecurity. We work fast, value ongoing learning, and we respect each employee as a unique individual. Knowing we all have different needs, our development and personal wellbeing programs are designed to give you choice in how you are supported. This includes our FLEXBenefits wellbeing spending account with over 1,000 eligible items selected by employees, our mental and financial health resources, and our personalized learning opportunities - just to name a few!

At Palo Alto Networks, we believe in the power of collaboration and value in-person interactions. This is why our employees generally work full time from our office with flexibility offered where needed. This setup fosters casual conversations, problem-solving, and trusted relationships. Our goal is to create an environment where we all win with precision.

Job Description

Your Career

We are seeking a Principal Software Engineer with deep expertise in designing, building, and scaling AI-powered platforms. In this role, you will help shape the future of our agentic platform by leveraging advanced machine learning (ML) techniques to tackle complex, large-scale challenges and deliver impactful customer experiences.

Lead the design, prototyping, and productionization of AI agent systems that solve complex user and business problems in cybersecurity applications. You’ll own agent architectures end-to-end—from planning and orchestration to evaluation, deployment, and observability—while mentoring engineers and shaping our AI strategy.

This role is located at our Santa Clara Headquarters Campus 3 days a week.

Your Impact

• Design & build agentic systems: Architect workflows and POCs using frameworks such as Google ADK and LlamaIndex; implement tool use, function calling, and multi-step planning.
• Retrieval & reasoning: Develop RAG pipelines (indexing, retrieval, reranking), code-interpreter/tool execution flows, and robust context management.
• Model evaluation: Define evaluation suites for performance, efficiency, safety, and business alignment; analyze latency, quality, and cost trade-offs.
• Scale & reliability: Deploy models and agents to production; build scalable ML pipelines for batch and real-time/streaming use cases; implement monitoring and guardrails.
• Platform & CI/CD: Drive end-to-end delivery with modern CI/CD; automate testing, rollout, and experiment tracking.
• Collaboration & leadership: Partner with ML engineers, data scientists, and product to deliver roadmaps; mentor teammates and lead technical design reviews.
• Documentation & communication: Maintain clear specs and decision records; communicate complex concepts to technical and non-technical audiences.
• Strategy & incubation: Contribute to AI product vision; incubate new AI initiatives and design microservices-based solutions on GCP.

Qualifications

Your Experience 

Basic qualifications

• 8+ years in ML, data/analytics, and software engineering with production experience.
• Strong coding skills in Python and proficiency with SQL, including performance/scalability optimization.

Preferred qualifications

• End-to-end experience designing and deploying RAG systems (indexing strategy, retrieval optimization, reranking).
• Expertise with LLMs and fine-tuning techniques (e.g., LoRA/QLoRA), prompt/agent design, and function-calling patterns.
• Familiarity with Google ADK (agents, long-term knowledge/memory) and LlamaI