About the role
<p><strong>About AppGate</strong></p><p>AppGate secures and protects an organization's most valuable assets with its high performance Zero Trust Network Access (ZTNA) solution. AppGate is the only direct-routed ZTNA solution built for peak performance, superior protection and seamless interoperability. AppGate safeguards Fortune 500 enterprises worldwide. Learn more at appgate.com. </p><p></p><p><strong>About the Role</strong></p><p>We’re looking for an <strong>Observability Engineer</strong> (Senior/Staff level) who has shipped distributed tracing systems, designed high-cardinality pipelines, and knows OpenTelemetry inside and out. You will own the end-to-end design and implementation of the AppGate observability fabric — from telemetry SDKs in our clients and gateways, to the LogForwarder pipeline, to customer-side integrations.</p><p>You’ll make the foundational technical decisions — transport protocols, sampling strategies, schema design, correlation models — that determine whether our platform scales gracefully to hundreds of millions of events per day. This is a builder’s role with a strategist’s reach.</p><p></p><p><strong>Key Responsibilities</strong></p><p>Your engineering work will directly enable next-generation capabilities, including:</p><p>• <strong>OpenTelemetry-Native Telemetry Fabric: </strong>Logs and distributed traces from clients, controllers, gateways, and connectors — all correlated by session, user, device, and trace ID across the full ZTNA flow.</p><p>• <strong>High-Cardinality Data Pipeline: </strong>An OTLP-based ingestion and routing layer engineered for 100M+ events per day, with attribute filtering, redaction, and tail-sampling.</p><p>• <strong>End-to-End Distributed Tracing: </strong>Span hierarchies decomposing login and session establishment across posture checks, policy decisions, TLS handshakes, and entitlement resolution — turning hours of triage into seconds.</p><p>• <strong>On-Demand Packet Capture: </strong>Admin-triggered PCAP coordinated across client and gateway, with the workflow fully observable through OTel logs and traces.</p><p>• <strong>AI-Ready Foundation: </strong>Structured, semantically rich telemetry that future LLM-based incident analysis agents can reason over. The schema you design today is the substrate for Phase 3.</p><p>• <strong>Architect the Observability Platform: </strong>Define telemetry schema, correlation model, transport, and sampling strategies spanning client devices, controllers, and gateways.</p><p>• <strong>Build the Telemetry SDKs and LogForwarder: </strong>Instrument AppGate components with OpenTelemetry and implement the enrichment, redaction, batching, and tail-sampling pipeline that scales horizontally under load.</p><p>• <strong>Validate at Customer Scale: </strong>Test in lab environments matching our largest deployments — hundreds of sites, tens of thousands of concurrent sessions — and hunt down cardinality explosions and pipeline backpressure before customers see them.</p><p>• <strong>Drive Integration Standards: </strong>Own the OTLP, Prometheus, and JSON-log compatibility surface and validate ingestion into Datadog, Splunk, Nexthink, and Elastic.</p><p>• <strong>Raise the Engineering Bar: </strong>Establish patterns and review practices the Data + AI team builds on. Mentor engineers and grow the observability discipline inside AppGate.</p><p>• <strong>Collaborate Cross-Functionally: </strong>Work directly with product, R&D, and marquee customers in defense and critical infrastructure to shape requirements and deliver outcomes that matter.</p>