AI-Factory Observability Principal

AI-Factory Observability Principal

 We are operating large-scale AI training and inference data centers, and we need an expert who can see the entire stack at once — from the chiller plant and switchgear to the GPU fabric and the Kubernetes scheduler. This role spans facilities/OT telemetry (cooling, power) and IT/AI infrastructure observability (compute, network, accelerators), unified by a single goal: complete, real-time, predictive visibility into how AI infrastructure consumes power, generates heat, moves data, and delivers compute.
You will design the observability platform that ingests signals from building and electrical systems, server and network fabrics, Kubernetes, and GPU/accelerator clusters — then apply AI/ML models on top of that telemetry to optimize utilization, predict failures, reduce energy cost, and surface insights operators can act on. You are equally comfortable reading a BACnet point list and a GPU NVLink topology, and you can explain to both facilities and platform teams why their data belongs in the same system.

 Observability architecture & strategy

• Define and own the end-to-end observability architecture covering metrics, logs, traces, and events across facilities and IT domains.
• Establish standards for instrumentation, telemetry pipelines, data retention, cardinality management, and a unified data model that lets power, thermal, network, and compute signals be correlated in one place.
• Design for scale: hundreds of thousands of time series per site, high-frequency power and thermal sampling, and GPU-cluster-level granularity.

Facilities & OT integration (BMS / EPMS)

• Integrate Building Management System (BMS) telemetry — CRAC/CRAH units, chillers, cooling loops, airflow, temperature/humidity, leak detection — into the central observability platform (BACnet, Modbus, MQTT, OPC-UA).
• Integrate Electrical Power Monitoring System (EPMS) data — switchgear, UPS, PDUs, busways, branch-circuit metering, generators — for real-time power draw, capacity, and quality monitoring (Modbus, DNP3, IEC 61850).
• Build correlated views of power and thermal behavior against compute workload so operators understand cause and effect (e.g., a training job's effect on rack power and inlet temperatures).
• Partner with facilities engineering on PUE, capacity planning, stranded-power recovery, and thermal optimization.

AI cluster & Kubernetes observability

• Architect observability for AI/GPU clusters — accelerator utilization, memory pressure, thermals, ECC/Xid errors, power capping, and job-level efficiency (e.g., via NVIDIA DCGM, accelerator telemetry exporters).
• Instrument Kubernetes environments running AI/ML workloads: cluster, node, pod, and workload metrics, scheduler behavior, GPU/accelerator allocation, and operator health.
• Provide visibility into training and inference pipelines — throughput, queue depth, checkpoint behavior, straggler detection, and cost-per-token / cost-per-training-step metrics.
• Surface noisy-neighbor, fragmentation, and underutilization patterns across multi-tenant clusters.

Network observability

• Design monitoring for high-performance data center fabrics, including the AI back-end network (RDMA, InfiniBand and/or RoCE Ethernet) and front-end/management networks.
• Capture fabric health, congestion, link errors, latency, and bandwidth utilization using streaming telemetry, SNMP, gNMI/gRPC, NetFlow/sFlow, and fabric managers (e.g., InfiniBand UFM).
• Correlate network behavior with distributed training performance to diagnose collective-communication bottlenecks.

AI/ML-driven optimization & insight (AIOps)

• Apply ML and AI models to the telemetry estate for anomaly detection, predictive maintenance, capacity forecasting, and automated root-cause analysis.
• Build models and pipelines that recommend (or automate) actions: dynamic cooling and power optimization, workload placement, power capping under thermal/electrical constraints, and failure pre-emption.
• Leverage LLMs and modern AI techniques to summarize incidents, accelerate root-cause investigation, query telemetry in natural language, and generate operator-facing insights from large volumes of logs and metrics.
• Establish the feedback loop where observability data trains the models that, in turn, optimize the infrastructure being observed.