KOEN
Read Original
#DevOps

Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale

Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale
01

Summary

How Netflix Solved 4-Hour Incident Investigations with an Ontology-Driven Knowledge Graph

Overcoming fragmented telemetry and LLM stochasticity with W3C ontology standards and deterministic AI agents at scale.

In massive streaming and microservice environments, fragmented metrics and inconsistent entity naming between tools hinder rapid incident response. Netflix addressed this by establishing an Observability Ontology using W3C standards and building a self-improving knowledge graph fed by LLM enrichers. This enables agentic AI solutions like AutoRCA to deliver ranked, evidence-backed root cause hypotheses instantly, saving hours of manual investigation.

  • 01Unified telemetry across 38 million events/sec into a single W3C RDF/OWL-based Observability Ontology.
  • 02Transformed stochastic LLM outputs into deterministic graph triples using strict type validation and SHACL schemas.
  • 03Utilized QuipuDB, a homegrown distributed RDF quad store built on Rust-based redb key-value store.
  • 04Built a Knowledge Flywheel that converts unstructured incident Slack discussions into structured triples.
  • 05Accelerated AutoRCA agent investigations by absorbing 1-2 hours of manual context assembly.

RECOMMENDATION

Highly recommended for DevOps and backend engineers managing complex microservices who face long incident investigation times. It provides invaluable architectural patterns for using knowledge graphs to anchor AI agents with deterministic context and eliminate hallucinations.

The Problem

At Netflix scale with 65 million concurrent streams and over 38 million real-time logging events per second, investigating operational regressions took around four hours due to extreme system complexity. Isolated telemetry, deployment, and alerting tools used inconsistent entity definitions, requiring human engineers to manually connect domain contexts across disparate systems.

The Solution

Netflix built an Observability Ontology using W3C standards (RDF, OWL, SHACL, SPARQL) and stored the operational knowledge graph in QuipuDB, a distributed RDF quad store built on redb. They implemented sequential enrichment pipelines to convert noisy data and LLM summaries into deterministic triples, integrated a Correlation Engine scoring causal chains, and built an agentic AI solution (AutoRCA) on top.

The Result

The system absorbed the one to two hours previously required to collect context and reconstruct cross-domain relationships, allowing engineers to immediately review ranked, evidence-backed hypotheses. Furthermore, human operational knowledge and discussion context trapped in Slack channels were transformed into structured triples, turning transient incident knowledge into reusable system intelligence.

Trade-off

Executing enrichment pipelines sequentially sacrificed wall-clock parallel speed to eliminate race conditions and enforce strict type ordering. Additionally, because AI-driven hypothesis ranking can occasionally be incorrect, human verification remains necessary before taking action.

03

Key Concepts

Concept · 01

Observability Ontology

A formal machine-readable specification defining operational entities (applications, services, incidents, deployments) and their relationships.

  • Built on W3C standards including RDF, OWL, SHACL, and SPARQL.
  • Organized into twelve namespaces, each owning one operational domain.
  • Gates every application entity against a software catalog of roughly 49,000 entries.
Concept · 02

Knowledge Flywheel

A self-improving architectural loop that continuously enriches system context through four phases: Observe, Enrich, Infer, and Adapt.

  • Encodes discovered incident cause-and-effect patterns into ontology triples.
  • Transforms repeating incident patterns into proactive detection rules and operational wisdom.
Concept · 03

AutoRCA

An internal agentic AI solution at Netflix that uses the knowledge graph as an interface to investigate regressions and present ranked root-cause hypotheses.

  • Coordinates specialized AI agents that communicate seamlessly via the ontology vocabulary.
  • Drastically reduces manual context assembly by presenting hypotheses attached with graph evidence.