KOEN
Read Original
#AI

Building Pinterest’s VLM Serving Stack on NVIDIA Dynamo

Building Pinterest’s VLM Serving Stack on NVIDIA Dynamo
01

Summary

Serving Images 85x Faster! How Pinterest Built a Next-Gen VLM Serving Stack with NVIDIA Dynamo and B200

Pinterest's architectural breakthrough dramatically cuts latency and TCO for high-scale, multi-turn multimodal inference.

Pinterest has redesigned its VLM serving infrastructure to power visual-heavy experiences like Pinterest Assistant. By substituting raw image pixels with precomputed projection embeddings and combining disaggregated serving with advanced KV caching, Pinterest successfully scales real-time serving to handle hundreds of images per request.

  • 01Adopted NVIDIA Blackwell B200 GPUs, reducing TCO and delivering more than 2x latency improvements compared to Hopper during benchmarking.
  • 02Implemented disaggregated prefill and decode serving to isolate heavy prefill computation from sequence generation.
  • 03Achieved up to 369x faster TTFT by passing precomputed PinCLIP embeddings instead of raw pixels, avoiding expensive runtime vision encoding.
  • 04Leveraged LMCache for multi-tier KV cache offloading across GPU, CPU DRAM, and local NVMe, alleviating high VRAM pressure during multi-turn chats.
  • 05Integrated AIPerf, a modular DAG-based benchmarking tool, to closely mimic complex agentic production traffic and prevent regressions.

RECOMMENDATION

Highly recommended for AI platform engineers and backend developers seeking to scale multimodal LLMs while tackling heavy image-processing overhead and severe KV cache limitations.

The Problem

Serving Vision-Language Models (VLMs) suffers from expensive and highly variable prefill overhead due to visual token processing, and severe KV cache memory pressure caused by large visual contexts in multi-turn interactions.

The Solution

Pinterest built a high-performance serving stack powered by NVIDIA Blackwell B200 GPUs and the NVIDIA Dynamo orchestration framework, implementing disaggregated prefill/decode serving, multi-tier KV cache offloading (LMCache), and precomputed PinCLIP projection embeddings to bypass real-time vision encoding.

The Result

Leveraging projection embeddings achieved an average of 85x (up to 369x) faster TTFT, 7.3x (up to 44x) faster E2E latency, and supported 25x more visual context. Multimodal KV-aware routing improved TTFT p99 by 1.42x on average, while B200 GPUs offered a 2x latency reduction over Hopper.

Trade-off

The larger precomputed embedding payloads created processing bottlenecks in the vLLM frontend, and managing consolidated artifacts (including model weights, configs, and projector weights) increased deployment complexity.

03

Key Concepts

Concept · 01

Disaggregated Prefill/Decode Serving

An inference architecture that splits the compute-heavy prefill phase and the memory-bandwidth-bound decode phase into separate hardware instances. This allows independent scaling of workers to maximize hardware efficiency and meet stringent latency SLOs.

  • Pinterest separated prefill (P) and decode (D) workers to explore the Pareto curve, significantly reducing Time-to-First-Token (TTFT).
Concept · 02

Projection Embeddings

A technique where precomputed vector embeddings are sent to the model instead of raw image files. A projection layer maps these embeddings directly into the native visual token space of the VLM, bypassing raw pixel decoding and online vision encoding.

  • By utilizing precomputed PinCLIP embeddings, Pinterest avoided image downloading, resizing, and online vision transformer computations.
  • This approach enabled requests with 250 images to reach latencies comparable to pixel-based requests carrying only 10 images.
Concept · 03

KV Cache Offloading & LMCache

A multi-tiered caching mechanism that offloads inactive or reusable KV cache states from limited GPU VRAM to larger, cheaper memory tiers like CPU RAM and local NVMe disks, retrieving them asynchronously when needed.

  • Integrated LMCache into vLLM to preserve low-latency generation during high-QPS, multi-turn conversational workloads.
  • Employed asynchronous prefetching and cache compression to maintain throughput without evicting high-token visual contexts.