KOEN
Read Original
#Backend

High-Throughput Graph Abstraction at Netflix: Part I

High-Throughput Graph Abstraction at Netflix: Part I
01

Summary

Netflix's Blueprint for Handling 10 Million Graph Ops Per Second

Deep dive into the specialized Graph Abstraction layer managing 650TB of real-time data at scale.

This article provides a masterclass on building high-throughput distributed graph systems. It demonstrates how Netflix leverages existing data abstractions to solve complex relationship queries without sacrificing performance or scalability. It details the intricate balance between consistency and latency in a massive-scale production environment.

  • 01OLTP-optimized architecture handling 10M operations per second.
  • 02Decoupling of edge links and properties to prevent Cassandra 'Wide Row' issues.
  • 03Asynchronous entropy repair using Kafka to ensure eventual consistency.
  • 04Strongly typed schemas used for query planning and data quality assurance.
  • 05Forward and reverse indexing strategy for efficient bidirectional traversals.

RECOMMENDATION

Highly recommended for backend architects and distributed systems engineers dealing with massive-scale graph data and latency constraints.

The Problem

Netflix needed an OLTP-centric graph architecture capable of supporting millions of operations per second with millisecond-level latency for real-time streaming and user experiences. Standard graph solutions often failed to meet both the high throughput and low latency requirements simultaneously.

The Solution

Netflix developed a 'Graph Abstraction' layer on top of its existing Key-Value and TimeSeries abstractions, utilizing a Property Graph model with strongly typed schemas. The architecture decouples edge links from property indexes and employs a Kafka-based asynchronous entropy repair mechanism.

The Result

The system now handles approximately 10 million operations per second across 650 TB of graph datasets with high cost-efficiency. It successfully powers critical use cases like real-time service topology and social graphs for gaming.

Trade-off

To prioritize performance, the system sacrifices atomic writes across multiple namespaces, requiring acceptance of eventual consistency and the implementation of complex background repair mechanisms.

03

Key Concepts

Concept · 01

Property Graph Model

A versatile graph data model where nodes and edges can hold associated properties.

  • Used to define dynamic relationships across entities in the Netflix ecosystem.
  • Strongly typed properties enable efficient filtering and consistent data exports.
Concept · 02

KV Abstraction (Key-Value)

A distributed key-value storage abstraction layer at Netflix serving as the real-time graph index.

  • Enforces Last-Write-Wins (LWW) semantics for conflict resolution.
  • Enables single-digit millisecond latency for node and property lookups.
Concept · 03

Entropy Repair

A robust mechanism to fix data inconsistencies across distributed indexes.

  • Utilizes Kafka-based retry mechanisms to ensure eventual consistency across namespaces.
  • Manages complex asynchronous edge deletions during concurrent updates.