KOEN
Read Original
#AI

In-House LLM Serving at Netflix

In-House LLM Serving at Netflix
01

Summary

Beyond Hosted APIs: How Netflix Scaled In-House LLM Serving

An in-depth look at how Netflix combined vLLM and Triton to achieve zero-downtime rollouts and high-throughput constrained decoding.

Netflix built an integrated, in-house LLM serving system on top of their existing production infrastructure, moving away from third-party hosting. This article details their key technical decisions—from engine selection to resolving unexpected CPU bottlenecks during production load.

  • 01Selected vLLM as the paved-path engine to speed up iteration times by avoiding tedious model compilation pipelines.
  • 02Decoupled model artifacts from frontend development by dynamically generating I/O specs through Triton's vLLM backend.
  • 03Introduced an OpenAI-compatible API alongside gRPC, enabling frictionless transition from external hosted models to in-house models.
  • 04Supported both Red-Black deployments for stable configurations and Versioned deployments for breaking schema changes.
  • 05Overcame serialization bottlenecks in vLLM V0 by rewriting custom logits processors to utilize batch-level C++ structures in vLLM V1.

RECOMMENDATION

Highly recommended for machine learning platform and DevOps engineers aiming to optimize GPU utilization, minimize external API dependency, or scale rule-based structured generation.

The Problem

Netflix aimed to run a full LLM serving stack in-house within their existing production environment to avoid hosted API costs and retain data privacy. They faced engineering hurdles in accelerating model compilation, enabling dynamic I/O specs, orchestrating zero-downtime rollouts, and resolving CPU bottlenecks during scaled, real-time constrained decoding.

The Solution

Netflix integrated vLLM into NVIDIA Triton as a backend, supported by a Java-based control plane and an OpenAI-compatible HTTP frontend. To eliminate constrained decoding bottlenecks, they migrated to vLLM V1 for better state tracking and rewrote their custom logits processor to operate on batch-level structures using multi-threaded C++ code.

The Result

The platform achieved seamless experimentation-to-production pathing with no-code API changes and established highly stable rollouts using Red-Black and Versioned deployment strategies. Upgrading to vLLM V1 and implementing the batched C++ logits processor kept tail latencies flat even as the batch size increased.

Trade-off

Utilizing Triton's vLLM backend creates version-coupling issues that require pinning matching service images, while custom execution logic still requires falling back to the complex Python backend. Additionally, Versioned deployments temporarily inflate GPU cost during model version transitions.

03

Key Concepts

Concept · 01

vLLM

An open-source LLM serving engine optimized for high-throughput and low-latency inference using efficient memory management techniques like PagedAttention.

  • Adopted as the default engine to load custom architectures rapidly and facilitate easy debugging for ML researchers.
  • Upgraded to V1 to leverage batch-level execution hooks for advanced constraint logic under heavy traffic.
Concept · 02

Triton Inference Server

An open-source inference serving platform from NVIDIA that streamlines model deployment across various frameworks on both CPUs and GPUs.

  • Served as the core compute engine of Netflix's Model Scoring Service, hosting vLLM to unify heterogeneous model serving.
Concept · 03

Constrained Decoding

A technique that restricts token generation in real-time by adjusting logits, forcing the model to produce outputs that conform strictly to a predefined schema or JSON format.

  • Implemented directly inside the decode loop to prevent malformed responses and bypass costly post-inference retries.
  • Developed using custom logits processor interfaces to supply dynamic state-machine masks for each request.