KOEN
Read Original
#DevOps

Mount Mayhem at Netflix: Scaling Containers on Modern CPUs

Mount Mayhem at Netflix: Scaling Containers on Modern CPUs
01

Summary

Why Did Netflix's Container Scaling Hit a Wall on Modern CPUs?

From Linux kernel VFS lock contention to hardware cache designs, uncovering Netflix's secrets to container runtime optimization.

This technical report explores how Netflix diagnosed system lockups during its container platform migration by diving deep into hardware-level details. It explains how NUMA and cache architectures of modern CPUs intersect with global locks to create performance bottlenecks, backed by empirical benchmarks.

  • 01Analysis of the risks of mount operations increasing exponentially with container image layers (50+)
  • 02Impact assessment of Linux VFS global lock contention causing system delays over 30 seconds
  • 03Benchmarking performance differences in lock contention handling between centralized (Intel) and distributed (AMD) cache architectures
  • 04Case study on improving lock contention latency by 20-30% by disabling Hyperthreading (HT)
  • 05A permanent fix by contributing to containerd upstream, optimizing mount logic from O(n) to O(1)

RECOMMENDATION

If operating large-scale container environments, optimize image layer counts and verify potential runtime performance degradation caused by hardware characteristics when using idmap features for security.

The Problem

Netflix encountered critical performance bottlenecks, including system stalls and 30-second health check timeouts, when launching many multi-layered containers simultaneously during its migration to a new container runtime (kubelet + containerd).

The Solution

The team identified global lock contention in the Linux kernel VFS as the root cause and optimized containerd by changing per-layer mount operations to a single parent-directory-based operation, reducing complexity from O(n) to O(1).

The Result

Eliminating the global lock contention enabled stable container scaling across various CPU architectures, recovered CPU resources previously wasted on lock spinning, and normalized container startup latencies.

Trade-off

Adopting idmap features for per-container user ranges improved security but paradoxically triggered a surge in kernel mount operations, requiring a careful balance between performance and security during the initial phase.

03

Key Concepts

Concept · 01

idmap mounts

A kernel feature that allows mapping UID/GID at the kernel level without changing file ownership, enabling safe file sharing across different user namespaces.

  • Introduced in Netflix's new runtime to enhance security isolation.
  • Caused excessive mount calls for every single layer in an image.
Concept · 02

NUMA (Non-Uniform Memory Access)

A memory design for multi-processor systems where each CPU has its own local memory but accesses remote memory with higher latency via an interconnect.

  • Amplified latencies during global lock synchronization on multi-socket instances like r5.metal.
  • Identified as a key factor in higher failure rates compared to single-socket instances.
Concept · 03

VFS Global Lock

A synchronization mechanism in the Linux Virtual Filesystem used for global operations like modifying the mount table.

  • Caused massive CPU spinning when thousands of concurrent mount requests competed for the same lock.
  • Optimized by reducing mount operations per container startup to an O(1) complexity.