KOEN
Read Original
#Backend

From Silos to Service Topology: Why Netflix Built a Real-Time Service Map

From Silos to Service Topology: Why Netflix Built a Real-Time Service Map
01

Summary

"No More 3 AM Troubleshooting Nightmares": How Netflix Visualizes Thousands of Service Dependencies in Real-Time

From eBPF to Graph Databases: The Architecture Behind a Living Map of Microservices

In a massive microservices environment, answering 'what connects to what' is both fundamental and notoriously difficult. Netflix solved this by building a real-time service topology based on actual traffic, moving beyond fragmented monitoring data. This article dives deep into how Netflix integrates multiple data layers and efficiently processes large-scale graph data to empower its engineering teams.

  • 01Capturing network flows at the kernel level using eBPF for comprehensive visibility without code modification
  • 02A 3-stage aggregation pipeline using Pekko Streams to resolve intermediaries like load balancers and reconstruct direct app-to-app paths
  • 03A multi-layer strategy that stores network, application, and request layers separately and merges them in real-time
  • 04Implementation of 'Time Travel' functionality to query dependencies and architectural changes at specific historical points
  • 05Utilization of a high-throughput graph database abstraction providing sub-second query response times at Netflix scale

RECOMMENDATION

Highly recommended for infrastructure and platform engineers in organizations where microservice complexity hinders incident response. Netflix's architecture serves as an excellent reference for those considering eBPF or graph databases to enhance distributed system visibility.

The Problem

In a complex architecture with thousands of microservices, it was extremely difficult to identify service dependencies and the blast radius during production incidents. Existing metrics, logs, and tracing tools provided fragmented views, forcing engineers to manually piece together the infrastructure map under high-pressure 3 AM scenarios.

The Solution

Netflix built a real-time Service Topology by integrating three complementary data sources: eBPF network flows, IPC metrics, and distributed tracing. They implemented a three-stage distributed aggregation pipeline using Apache Pekko Streams to resolve network intermediaries and utilized a high-throughput graph database to achieve sub-second query response times.

The Result

Engineers can now visually explore upstream/downstream dependencies and rapidly track failure propagation paths. Efficiency in incident investigation and architectural planning significantly improved through automated service tier verification and the ability to reconstruct historical topology states.

Trade-off

The approach balances different data source limits: eBPF lacks endpoint context, while IPC metrics and tracing can miss paths due to instrumentation gaps or sampling. Managing these three layers separately and merging them at query time introduces significant architectural complexity and storage overhead.

03

Key Concepts

Concept · 01

eBPF

A technology that allows running sandboxed programs in the Linux kernel to observe and control system behavior without changing kernel source code. It is widely used for high-performance networking and observability in cloud-native environments.

  • Used to capture raw network flow records, providing a ground truth for all connections regardless of application instrumentation.
Concept · 02

Service Topology

A visualization of the logical and physical connections and dependencies between microservices within a distributed system. It reflects real-time runtime traffic rather than static documentation.

  • Netflix implemented this as a 'living map' that updates continuously to reflect service deployments and changing traffic patterns.
Concept · 03

Apache Pekko Streams

A fork of Akka, it is a toolkit for building highly concurrent, distributed, and resilient message-driven applications on the JVM with built-in backpressure handling.

  • Served as the core engine for the distributed processing pipeline that aggregates millions of network records into a coherent graph.