KOEN
Read Original
#Backend

Netflix Conductor: The Next Chapter

Netflix Conductor: The Next Chapter
01

Summary

Inside Netflix Conductor 4.0: Rearchitecting Distributed Orchestration for Billions of Workflows

How Netflix eliminated lock contention and scaled workflow execution by 10x after pivoting from OSS to internal innovation.

After stepping back from open-source maintenance, Netflix rebuilt Conductor from the ground up to power over 420 million monthly workflow executions. By decoupling task evaluation into an asynchronous architecture and partitioning datastore models, Netflix eliminated synchronous lock contention while cutting tail latency. This article offers a deep engineering dive into building hyper-scalable distributed orchestration layers.

  • 01Expanded single workflow task capacity from 2,500 to 30,000 tasks (over 10x growth)
  • 02Eliminated Cassandra LWT locks by adopting a dual-partition (Pending/Terminal) state model resolved at the application layer
  • 03Achieved a 40% reduction in p99 workflow evaluation latency using Timestone exclusive queues
  • 04Introduced serverless managed tasks integrated with Titus container orchestration and Stratum functions
  • 05Implemented dynamic thread allocation auto-scaled by real-time queue depth

RECOMMENDATION

Essential reading for backend and infrastructure engineers designing high-throughput orchestration engines or struggling with database locks in microservice architectures.

The Problem

As Netflix Conductor scaled to process billions of quarterly executions, it encountered severe performance bottlenecks due to Dynomite in-memory store limitations, synchronous evaluation lock contention, and high memory pressure from monolithic workflow data structures.

The Solution

Netflix modernized its data plane by migrating execution data to Cassandra and deploying Timestone for queuing, while introducing Conductor 4.0 with a partitioned task model and an asynchronous, queue-driven evaluation engine.

The Result

The system increased maximum task capacity per workflow tenfold from 2,500 to 30,000 tasks, reduced p99 workflow evaluation latency by approximately 40%, and dropped failed lock acquisition attempts to virtually zero.

Trade-off

Netflix discontinued the public open-source project (Conductor OSS) to focus internal resources, increasing operational reliance on specialized internal infrastructure components such as Timestone, Titus, and Stratum.

03

Key Concepts

Concept · 01

Timestone

Netflix's high-throughput, low-latency priority queuing system built to replace DynoQueues. It provides built-in delayed queuing and background workflow repair capabilities.

  • Eliminated Dynomite in-memory datastore limits during heavy burst traffic.
  • Powers the exclusive queues that drive asynchronous workflow evaluation without lock contention.
Concept · 02

Partitioned Task Model

A data architecture that decouples workflow metadata from individual task payloads. The evaluation engine loads only lightweight blueprints and active task records instead of the entire execution tree.

  • Dramatically decreased evaluation memory footprints, enabling 10x task expansion per workflow.
  • Splits state into pending and terminal partitions to resolve concurrent write races without locking.
Concept · 03

Async Workflow Evaluation

An architectural pattern that decouples execution triggers from inline processing by enqueuing updates into dedicated exclusive queues processed sequentially by background evaluators.

  • Eliminated lock contention races caused by concurrent updates to the same workflow execution.
  • Reduced failed lock acquisition spikes to zero while significantly flattening p99 tail latency.