KOEN
Read Original
#Backend

TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads

TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads
01

Summary

How Netflix Scaled Petabyte-Scale Time-Series Storage by Reading Cold Data Directly from S3

The architectural journey from Kafka/Flink S3 streaming to native Cassandra backup reads

To cost-effectively manage multi-petabyte time-series datasets without breaking API compatibility, Netflix designed a multi-tiered storage platform. This post highlights their architectural evolution from an event-streamed Parquet S3 pipeline to a native Cassandra cold-tier solution that queries S3 snapshot backups using pluggable Memtables.

  • 01Achieved ~150ms p50 read latencies on cold data using Kafka/Flink and custom daily Parquet compaction
  • 02Developed ReadOnlyCompactionStrategy (ROCS) to eliminate SSTable overlap and lower read fan-out
  • 03Implemented a custom 'backupmemtable' via Cassandra 4.1 pluggable Memtables to read S3 backups directly
  • 04Optimized S3 read calls through localized index caching, chunk prefetching, and disabling unnecessary Read Repairs

RECOMMENDATION

Highly recommended for infrastructure engineers and database architects looking to build cost-effective tiered storage systems for petabyte-scale Cassandra workloads.

The Problem

Managing multi-petabyte time-series datasets in hot Cassandra storage tuned for low-millisecond access is cost-prohibitive for Netflix. As user demand for longer data retention grows, storing rarely accessed historical data in hot clusters creates severe financial and operational overhead.

The Solution

Netflix first implemented a streaming pipeline using Apache Kafka and Flink to persist data as Parquet files in S3 with daily compaction jobs. To eliminate streaming infrastructure costs, they evolved to a native Cassandra cold-tier architecture using Cassandra 4.1 pluggable Memtables and ReadOnlyCompactionStrategy (ROCS) to read directly from S3 snapshot backups.

The Result

The new architecture eliminated secondary streaming infrastructure and S3 data transfer fees while reducing Cassandra EBS storage footprint. It serves cold read queries with a stable p50 latency of ~150 ms and p99 latency of ~500 ms.

Trade-off

Reading directly from Cassandra S3 backups introduces an asynchronous initialization phase to download index metadata to local nodes. Additionally, reliability features like Read Repair and Speculative Retries must be disabled for immutable backup reads.

03

Key Concepts

Concept · 01

ReadOnlyCompactionStrategy (ROCS)

A custom Cassandra compaction strategy designed for immutable data that aligns SSTables non-overlappingly along token ranges.

  • Applied once a time slice becomes immutable to eliminate token overlaps between SSTable files.
  • Dramatically reduces read fan-out and tail latencies across all storage tiers.
Concept · 02

Pluggable Memtable

A feature introduced in Apache Cassandra 4.1 that allows developers to plug in custom Memtable implementations to redirect write and read paths.

  • Used to construct 'backupmemtable', routing queries for cold data directly to Cassandra S3 backups.
  • Downloads index and summary components locally to perform query planning before fetching data chunks from S3.
Concept · 03

Parquet Compaction Job

A daily batch process that merges small streaming Parquet files into consolidated, sorted Parquet files optimized for lookups.

  • Uses small (4–8 MB) row groups and Bloom filters cached in EVCache to skip unnecessary S3 reads.
  • Notifies the TimeSeries control plane upon completion so hot storage slices can be safely truncated.