KOEN
Read Original
#Backend

How and Why Netflix Built a Real-Time Distributed Graph: Part 3 — Querying the graph with gRPC…

How and Why Netflix Built a Real-Time Distributed Graph: Part 3 — Querying the graph with gRPC…
01

Summary

How Netflix Queries Billions of Graph Edges in Under 100 Milliseconds

Designing an ultra-low latency distributed graph execution engine using gRPC and async breadth-first traversal

This article reveals Netflix's design for the serving layer of its Real-Time Distributed Graph (RDG), which processes complex multi-hop queries in sub-100ms. By replacing depth-first traversal with async breadth-first execution and adopting streaming adjacency lists, Netflix achieves high performance and safety under massive scale.

  • 01Adopted Breadth-First Traversal (BFS) instead of DFS to eliminate sequential network latency overhead by batching lookups in parallel at each graph level.
  • 02Replaced the traditional thread-per-request model with fully asynchronous pipeline composition, managing thousands of requests with just 16 to 24 thread pools.
  • 03Streamed massive adjacency lists in batches of 100 instead of materializing them as giant blobs, applying early filtering to keep the memory footprint low.
  • 04Achieved a 70-80% cache hit rate using EVCache by selectively targeting stable node properties while intentionally bypassing volatile data.
  • 05Implemented an opt-in, fail-open enrichment layer that fetches external metadata in parallel and gracefully returns graph results even if external services fail.

RECOMMENDATION

Highly recommended for backend architects, distributed systems engineers, and API designers who need to retrieve heavily connected, large-scale data with strict latency guarantees.

The Problem

While Netflix built a Real-Time Distributed Graph (RDG) containing billions of nodes and edges, they faced a distinct set of challenges in the serving layer. They had to support diverse access patterns, from high-volume security lookups to deep exploratory queries, while maintaining a sub-100ms latency without accumulating sequential network hop overhead.

The Solution

To solve this, the execution engine orchestrates breadth-first traversal to expand graph levels in parallel and uses asynchronous composition built on a small, dedicated thread pool (16-24 threads) to prevent blocking on I/O. Furthermore, the engine processes adjacency lists as streams to avoid over-fetching and selectively caches frequently accessed, stable node properties via EVCache.

The Result

Through this design, complex multi-hop queries, such as a 2-hop viewing history traversal, are resolved in under 100ms. Thousands of concurrent queries are managed safely and efficiently without thread exhaustion, achieving a 70-80% cache hit rate on selective node lookups.

Trade-off

The breadth-first approach requires holding each level of the graph frontier in memory simultaneously, scaling costs with breadth rather than depth. To mitigate memory pressure, strict per-edge-type limits must be applied at each hop, and the system relies on eventual consistency rather than strong consistency to achieve high throughput.

03

Key Concepts

Concept · 01

Breadth-First Traversal

A graph traversal method that explores all nodes at the present depth level before moving to the nodes at the next depth level, minimizing network roundtrips in a distributed system.

  • Used to batch requests across multiple parallel paths, such as fetching viewing histories for all profiles simultaneously.
  • Trading off memory usage to gain parallel latency advantages, bounded by per-edge limits.
Concept · 02

Adjacency List Streaming

A method of retrieving edge connections for a node as an active stream of batches rather than a monolithic payload, allowing early termination.

  • Handles nodes with high fan-outs (e.g., highly active user viewing logs) without memory spikes.
  • Enables on-the-fly filtering and limit enforcement directly on incoming data chunks.
Concept · 03

Asynchronous Composition

An execution model where I/O operations are chained non-blockingly, returning threads back to the execution pool while waiting for network responses.

  • Allows Netflix to scale concurrent queries across tiny thread pools of 16-24 threads.
  • Orchestrates graph lookups, caching, and external service enrichments concurrently.