KOEN
Read Original
#DevOps

Reliable GPU compute for large training & inference jobs

Reliable GPU compute for large training & inference jobs
01

Summary

How Netflix Built Resilient GPU Infrastructure That Survives Hardware Failures

Inside the 4-layer architecture elevating Netflix's GPU availability to over 99.95%

To guarantee seamless AI recommendations and content production tools, Netflix built an automated GPU reliability platform capable of detecting and remediating fine-grained hardware failures without human intervention. By layering Node Problem Detector, custom DCGM fallback logic, and Karpenter auto-repair, Netflix transformed flaky hardware into predictable capacity. This article highlights practical approaches for operating large-scale distributed training and inference workloads.

  • 01Fallback mechanism reading kernel dmesg directly when GPU driver diagnostic tools hang
  • 02Graded remediation matching fault severity with job kills, GPU resets, reboots, or full instance replacements
  • 03Customized Karpenter auto-repair logic with tolerance windows and disruption caps to prevent cascading teardowns
  • 04Achieved >99.95% GPU availability and boosted job success rate from ~98% to over 99.5%

RECOMMENDATION

Highly recommended for infrastructure and MLOps engineers managing large GPU clusters or distributed training pipelines. It offers actionable architecture patterns for combining Kubernetes Node Problem Detector with custom remediation policies.

The Problem

In Netflix's large-scale ML model training and inference, a single faulty GPU or network link across hundreds of nodes can cause gang-scheduled jobs to slow down or fail completely. Basic cloud instance health checks and driver response checks failed to detect silent defects such as memory errors, reduced fabric bandwidth, or frozen GPU drivers.

The Solution

Netflix introduced a four-layer reliability architecture encompassing hardware detection, scheduler auto-repair, training checkpointing, and observability. They implemented two-tier health checking with Kubernetes Node Problem Detector, NVIDIA DCGM, and EFA, alongside custom Karpenter-driven graded remediation to automatically resolve faults ranging from GPU resets to full node replacements.

The Result

The platform consistently achieved >99.95% GPU node availability and raised job success rates from approximately 98% to above 99.5%.

Trade-off

Holding node scheduling capacity until checks settle slightly increases node onboarding latency, and care must be taken to configure disruption caps so buggy health checks do not trigger unintended cluster-wide node tear-downs.

03

Key Concepts

Concept · 01

Node Problem Detector (NPD)

An open-source Kubernetes daemon that monitors node-level hardware and OS issues and translates them into Kubernetes Node Conditions.

  • Deployed on every Netflix GPU instance as a node-level agent running checks every 30 seconds.
  • Translates DCGM metrics and EFA network adapter statuses into actionable Kubernetes Node Conditions.
Concept · 02

Graded Remediation

A recovery strategy that maps detected fault types to tailored actions ranging from soft GPU resets to full instance termination.

  • Clears software wedges and transient device faults in minutes via GPU resets or system reboots instead of slow replacements.
  • Escalates persistent hardware faults to node evacuation, draining, and Karpenter instance replacement.
Concept · 03

NVIDIA DCGM (Data Center GPU Manager)

A suite of tools provided by NVIDIA for monitoring and diagnosing GPU health, memory errors, and thermals in data center environments.

  • Used to query driver error codes, uncorrectable memory errors, and interconnect integrity.
  • Augmented with kernel dmesg parsing to ensure unhandled driver freezes do not hide fatal GPU errors.