KOEN
Read Original
#Backend

Scaling Airbnb’s identity graph with a unified knowledge graph infrastructure

Scaling Airbnb’s identity graph with a unified knowledge graph infrastructure
01

Summary

Scaling to 7 Billion Nodes: Airbnb's Blueprint for High-Performance Graph Infrastructure

How Airbnb achieved a 93% latency reduction by moving from PaaS to an optimized JanusGraph-DynamoDB stack.

This article explores Airbnb's journey in building a unified knowledge graph infrastructure to overcome the limitations of third-party vendors. It details the architectural shift toward storage-compute separation and the deep engine-level optimizations required for massive-scale identity resolution.

  • 01Leveraged storage separation with JanusGraph and DynamoDB for independent scaling.
  • 02Optimized Gremlin query planning to handle high-fanout, multi-hop traversals.
  • 03Implemented parallel execution in the JanusGraph core to reduce P99 latencies.
  • 04Custom transaction strategies using DynamoDB conditional writes for data integrity with low overhead.
  • 05Enhanced observability through internal forks supporting distributed tracing.

RECOMMENDATION

A must-read for backend and infrastructure engineers scaling graph-based systems. The insights on query rewriting and storage separation are highly applicable to large-scale data platforms.

The Problem

Airbnb's identity graph reached 7B nodes and 11B edges, causing scalability limits with PaaS solutions, high long-tail latency (P95/P99) for 4-8 hop queries, and operational instability.

The Solution

Built an internally managed knowledge graph platform using JanusGraph with DynamoDB for storage and OpenSearch for indexing. Implemented optimizations like custom transaction strategies using conditional writes and client-side Gremlin query rewriting.

The Result

Achieved 32-93% lower latency across query patterns and reduced end-to-end P99 latency by ~50%. Scaled write QPS by 10x compared to the previous vendor solution while eliminating manual reboots.

Trade-off

Shifting from a managed PaaS to an internal infrastructure requires significant upfront engineering investment and ongoing operational responsibility for maintaining the graph engine and storage integration.

03

Key Concepts

Concept · 01

JanusGraph

A scalable, distributed graph database engine built on Apache TinkerPop that supports various storage backends.

  • Served as the core computation engine for Airbnb's internal graph platform.
  • Enabled expressive traversal queries using the Gremlin query language.
Concept · 02

Storage Separation

An architectural pattern that decouples the graph processing logic from the underlying data persistence layer.

  • Allowed Airbnb to use AWS DynamoDB for reliability while iterating quickly on graph features.
  • Enabled independent scaling of storage capacity and query processing power.
Concept · 03

Gremlin Query Rewriting

The process of optimizing graph traversal queries by restructuring steps to better align with the engine's query planner.

  • Removed heavy 'Path' steps to prevent slow, non-batched backend queries.
  • Replaced side-effect steps with optimized conditional queries to minimize computation overhead.