Principal Datadog Engineer

Responsibilities

We are seeking a skilled Principal Datadog Engineer responsible for driving the strategic vision, architecture, and execution of engineering projects.  This role combines deep technical expertise with leadership and collaboration to ensure the success of complex initiatives.

Principal Datadog Engineers operate at a high level of autonomy, influencing organizational strategies and policies. They oversee multiple projects simultaneously, manage broader teams, and set the technical vision for their domain.  Unlike senior engineers who focus on execution within a specific project scope, Principal Engineers shape the direction of entire departments or organizations. They also engage with external stakeholders to represent the company's technical expertise and create formal networks for coordination among various groups.

What you will do:

• Define and implement high-level engineering strategies and system architectures that align with organizational goals. 
• Develop scalable observability strategies using Datadog for applications, infrastructure, cloud services, and security monitoring.
• Architect dashboards, monitors, and alerting frameworks tailored to business and operational requirements.
• Lead the design, development, and delivery of large-scale, complex engineering projects while ensuring quality and scalability.  
• Identify inefficiencies in workflows and implement tools, methodologies, or processes to enhance productivity and reliability. 
• Establish best practices for metric collection, log ingestion, tracing, and anomaly detection.
• Optimize cost management and usage efficiency of Datadog licenses and features.
• Address critical technical challenges by providing innovative solutions and guiding teams through implementation.
• Work on issues where analysis of situations or data requires an evaluation of intangibles.
• Serve as escalation point for Datadog-related performance or monitoring issues.
• Perform root-cause analysis using Datadog dashboards, traces, and logs to identify and resolve system issues quickly.
• Mentor and coach engineering teams, fostering a culture of excellence and continuous learning.
• May supervise the activities of others, providing guidance to ensure alignment with organizational goals. Conduct training, documentation, and workshops to upskill engineering teams in Datadog usage.
• Partner with product managers, stakeholders, and other departments to align technical solutions with business objectives. 
• Create formal networks involving coordination among groups, including key decision-makers. Partner with DevOps, SRE, Cloud, Security, and Application teams to embed observability into daily operations.
• Serve as the primary decision-maker for technical directions, balancing short-term needs with long-term goals. Exercise autonomy in critical decision-making processes.
• Will act as a trusted advisor and technical authority, enabling teams to maximize the value of Datadog for performance monitoring, application observability, infrastructure health, security insights, and operational excellence.
• Translate complex technical concepts into actionable insights for non-technical stakeholders. 
• Serve as an external spokesperson for the organization, representing the company's technical expertise.

Qualifications

Required Qualifications:

• Bachelor’s Degree and 8 years of experience or 9 years of experience and a HS Degree/Diploma.
• Must be willing to get/obtain the Datadog Fundamentals Certification
• Strong development background with hand on experience with Python, Bash, PowerShell, YAML and proficiency in building complex lambda functions / shell scripts for end-to-end automation.
• Hands-on experience enabling Datadog infrastructure monitoring and application performance monitoring.
• Experience configuring Datadog for synthetic tracing.
• Proven experience as a Cloud DevOps and Automation Engineer.
• Strong analytical skills to assess risks and opportunities in technical projects, especially when evaluating intangibles.
• Exceptional communication skills for effective collaboration across diverse teams and external representation of the organization.
• Strategic thinking to anticipate future challenges and opportunities, particularly in addressing future concepts, products, or technologies.
• Ability to inspire teams by setting clear goals and leading by example, demonstrating expertise in the field.
• Must be a US Citizen.
• Must be able to obtain and maintain a Public Trust 6C clearance.

Preferred Qualificatinos:

• Active Public trust
• Act