KOEN
Read Original
#Backend

Partition Finalization in Pinterest’s Next-Generation DB Ingestion Framework

Partition Finalization in Pinterest’s Next-Generation DB Ingestion Framework
01

Summary

How Pinterest Solved Data Completeness in Stream-Based DB Ingestion using Flink & Iceberg

Unlock seamless downstream batch processing with a metadata-driven, non-regressing partition finalization watermark.

This article details Pinterest's transition from batch to real-time streaming CDC ingestion and how they tackled the data completeness challenge. By extending the Flink-to-Iceberg sink with Accumulators and CommitProcessors, they built a self-contained metadata layer to track event-time progress. The resulting system enables robust configuration of completeness-latency trade-offs, allowing downstream jobs to safely schedule execution based on table properties.

  • 01Solves the fundamental data completeness problem for downstream batch consumers when transitioning to stream ingestion.
  • 02Eliminated the need for separate tracking infrastructure by leveraging native Iceberg table properties and Flink customization.
  • 03Utilized t-digest sketches to maintain accurate percentile estimates in under a few hundred bytes per commit.
  • 04Designed a seamless metadata propagation flow from CDC tables to downstream base tables via Spark upsert jobs.
  • 05Provides customizable policies allowing pipeline owners to fine-tune the balance between data freshness and completeness.

RECOMMENDATION

Highly recommended for data engineers building real-time CDC or event streams where downstream batch engines require reliable, low-overhead 'safe-to-read' indicators to schedule processing.

The Problem

While the legacy batch system processed complete database dumps at once, making partition finalization trivial, migrating to the next-generation streaming CDC framework introduced event-time lag, making it difficult to determine when a time partition is complete enough to read without missing updates.

The Solution

Pinterest introduced two generic extension points (Accumulators and CommitProcessor) into the Flink-to-Iceberg sink to collect event-time statistics via t-digest sketches during checkpoints, and applied a non-regressing watermark algorithm immediately after each commit to write the finalization status as an Iceberg table property.

The Result

By storing self-describing metadata of only a few hundred bytes, downstream batch jobs can query the partition finalization watermark using sensors without external services, and the signal is successfully propagated from CDC tables to base tables via Spark upserts.

Trade-off

A fundamental trade-off exists between data completeness and processing latency; choosing aggressive percentile policies for lower latency can cause extremely delayed data to arrive after partition finalization, which triggers alerts but risks missing updates.

03

Key Concepts

Concept · 01

t-digest

A highly efficient, mergeable sketch algorithm used to approximate probability distributions and percentiles with high precision under strict memory constraints.

  • Used to compact event-time distributions from parallel Flink writer subtasks into a small, fixed space.
  • Enables order-independent merging of subtask statistics at checkpoint commit time.
Concept · 02

Iceberg Snapshot Summary

A metadata area within Apache Iceberg snapshots that stores operation summaries and properties associated with each commit.

  • Serves as a self-describing, zero-external-dependency store for the collected event-time statistics on each commit.
Concept · 03

Non-regressing Watermark Algorithm

An algorithmic logic that progresses a logical watermark monotonically forward, ensuring that once a time boundary is marked complete, it never regresses.

  • Guarantees that downstream batch sensors read stable finalization signals even if late-arriving records appear.
  • Triggers user alerts when delayed data violates the finalized boundary.