KOEN
Read Original
#AI

Democratizing Machine Learning at Netflix: Building the Model Lifecycle Graph

Democratizing Machine Learning at Netflix: Building the Model Lifecycle Graph
01

Summary

Tired of Black Box Models? How Netflix Connected Thousands of ML Assets into a Single Graph

Unifying fragmented infrastructure to track ML lineage and business impact across domains.

This article explores Netflix's journey in building a Metadata Service (MDS) to solve the complexity of its ML ecosystem. It details how they used a graph-based approach to make every ML asset discoverable and understandable across the company.

  • 01Unified addressing for all ML assets using URI-based AIP identifiers
  • 02Real-time metadata streaming and enrichment via Kafka and SNS/SQS
  • 03Complex graph traversal and inference using Datomic's immutable fact model
  • 04Unified search and filtering across the entire ML catalog via Elasticsearch
  • 05Automated relationship mapping between pipelines and A/B test configurations

RECOMMENDATION

Highly recommended for platform engineers struggling with fragmented data infrastructure or senior developers looking to design large-scale MLOps systems.

The Problem

As Netflix expanded ML across domains like Personalization, Studio, and Payments, fragmented tech stacks turned models into 'black boxes,' hindering cross-team collaboration and asset discovery.

The Solution

Netflix built a Metadata Service (MDS) that implements a Model Lifecycle Graph using a URI-based addressing scheme, Kafka-based ingestion, Datomic for graph relationships, and Elasticsearch for search.

The Result

By connecting ML assets into a navigable graph, Netflix enabled real-time visibility into lineage and impact—from upstream data sources to downstream A/B tests—fostering cross-domain asset reuse.

Trade-off

The hydration process places additional read load on source systems, requiring deliberate rate limiting, and the asynchronous enrichment introduces a short delay in relationship visibility.

03

Key Concepts

Concept · 01

Model Lifecycle Graph

A data structure that visualizes and connects the entire ML lifecycle, including pipelines, datasets, models, and A/B tests.

  • Enables cross-domain collaboration by connecting siloed infrastructure
  • Provides transparency into model lineage and the impact of changes
Concept · 02

Datomic

An immutable, facts-based database optimized for complex joins and navigational queries across entity relationships.

  • Serves as the system of record for MDS graph relationships
  • Handles relationship-heavy, navigational queries spanning multiple hops in the graph
Concept · 03

Entity Enrichment

The process of fetching full metadata from source systems based on minimal event triggers to build a complete view of an entity.

  • Decouples event streaming from state consistency by always reading from the source of truth
  • Standardizes heterogeneous data into a unified entity model with global IDs