KOEN
Read Original
#Backend

DataJunction as Netflix’s answer to the missing piece of the modern data stack

DataJunction as Netflix’s answer to the missing piece of the modern data stack
01

Summary

How Netflix Solved Scattered Metrics with DataJunction Semantic Layer

Inside Netflix's architecture for scalable, trustworthy metrics across experimentation and analytics

This article highlights how Netflix tackled complex and fragmented metric definitions in its experimentation platform by leveraging DataJunction, an open-source semantic layer. Powered by a SQL parsing engine and a graph-based metadata store, DataJunction abstracts metric logic from compute engines. It offers an inspiring roadmap for organizations striving to unify data modeling, analytics, and AI context.

  • 01Decoupled metric definitions from statistics execution engines for effortless platform migrations
  • 02Leveraged a connected graph model to store relational metadata for views and queries
  • 03Executed an API-first and graph-first strategy under resource constraints
  • 04Integrated multidimensional cubes with Druid and Superset for transparent lineage and fast querying

RECOMMENDATION

Highly recommended for Data Engineers, Analytics Engineers, and Data Architects facing metrics fragmentation. Ideal for teams planning to implement a unified semantic layer or integrate business context into LLM data stacks.

The Problem

At Netflix, metric definitions within the experimentation platform were scattered and overly complex, requiring deep tribal knowledge and custom engineering workflows that took weeks to onboard new metrics.

The Solution

Netflix adopted DataJunction (DJ), an open-source semantic layer with a robust SQL parsing and generation engine operating on a connected metadata graph. They separated metric definitions from the execution engine by taking an API-first approach and employing multidimensional cubes.

The Result

Centrally governed metric definitions significantly simplified metric onboarding and ad-hoc troubleshooting, while enabling seamless integrations with analysis tools like Druid and Superset as well as internal LLMs.

Trade-off

Due to initial resource constraints, feature sets were deliberately limited, prioritizing the core semantic SQL engine and API over standard UI and non-essential auxiliary tooling.

03

Key Concepts

Concept · 01

DataJunction (DJ)

An open-source semantic layer platform that standardizes metric and dimension definitions through SQL parsing and generation. It acts as a central metadata graph layer across diverse data tools.

  • Introduced as the central hub to resolve metric definition fragmentation in Netflix's experimentation platform
  • Parses input SQL to build metadata graphs and dynamically generates optimized SQL serving layer queries
Concept · 02

Semantic Layer

An architectural abstraction layer between raw data stores and end-user consumption tools that provides consistent business definitions. It decouples metric logic from underlying compute platforms.

  • Eliminated tribal knowledge requirements and engineering friction during metric onboarding
  • Serves as the foundation for providing accountable metrics context to internal LLM applications
Concept · 03

Multidimensional Cube

An analytical data framework encapsulating arbitrary aggregations of shared metrics and dimensions into unified serving units for fast analysis.

  • Adopted as a first-class feature in DataJunction to streamline metric browsing and data retrieval
  • Materialized directly into internal Druid clusters to power real-time data exploration dashboards