KOEN
Read Original
#Backend

The Evolution of Cassandra Data Movement at Netflix

The Evolution of Cassandra Data Movement at Netflix
01

Summary

How Netflix Moves 3PB of Cassandra Data Daily While Saving Millions in Costs

The evolution from monolithic Casspactor to a scalable, S3-native data movement engine.

This article explores how Netflix redesigned its Cassandra-to-Iceberg data pipeline to overcome scalability limits. It details the technical shifts in data retrieval and the sophisticated migration strategies like the 'Decider Pattern' that ensured zero downtime for thousands of internal data pipelines.

  • 01Eliminated fragile service dependencies by making S3 the single source of truth for backups.
  • 02Solved OOM issues for skewed partitions by moving mutation processing to Spark Executors.
  • 03Saved millions in storage costs by operating directly on Spark DataFrames without intermediate tables.
  • 04Achieved zero-impact migration using the 'Decider Pattern' for transparent system swaps.
  • 05Enabled robust Time Travel capabilities for auditing, debugging, and disaster recovery.

RECOMMENDATION

Highly recommended for Data Engineers and Backend Architects looking for proven strategies in large-scale data migration and high-throughput pipeline optimization.

The Problem

Netflix's legacy data movement engine, Casspactor, faced issues with fragile metadata dependencies, out-of-memory errors on skewed partitions, and high storage costs due to the creation of multiple intermediate Iceberg tables.

The Solution

They introduced a new layered architecture built on Apache Cassandra Analytics that reads directly from S3 backups, processes data as Spark DataFrames, and utilizes a 'Connector Factory' model to handle various data abstractions efficiently.

The Result

The migration resulted in millions of dollars in annual cost savings by eliminating intermediate tables, improved stability for large partitions, and added critical 'Time Travel' functionality for reproducing past data states.

Trade-off

While reducing dependencies on internal metadata services, the new system's reliance on direct S3 metadata requires high storage consistency, and the migration necessitated significant engineering effort for rigorous shadow validation.

03

Key Concepts

Concept · 01

Apache Cassandra Analytics

An open-source library that enables bulk reading and writing of Cassandra data without impacting the performance of the production cluster.

  • Used as the foundation to read SSTables directly from S3 and translate them into Spark DataFrames.
Concept · 02

Decider Pattern

An architectural pattern that dynamically determines which implementation to use at runtime based on configuration.

  • Enabled Netflix to swap the legacy engine with the new one transparently for downstream users without code changes.
Concept · 03

Shadow Testing

A validation technique where new systems are run in parallel with legacy systems using real production traffic to compare outputs.

  • Crucial for establishing trust by proving 100% row-by-row consistency between the old and new data movement engines.