KOEN
Read Original
#AI

Scaling Recommendation Systems with Request-Level Deduplication

Scaling Recommendation Systems with Request-Level Deduplication
01

Summary

How Pinterest Boosted Recommendation Serving Throughput by 7x

A strategic engineering approach to scaling models 100x while keeping infrastructure costs in check.

This article explores Pinterest's 'request-level deduplication' technique used to manage massive recommendation models. It details how eliminating data redundancy across storage, training, and serving leads to breakthrough performance gains and solves critical architectural challenges.

  • 01Up to 50x storage compression via Apache Iceberg and request-based sorting
  • 02Restoring model quality with SyncBatchNorm by addressing disrupted IID statistics
  • 03Preventing retrieval quality degradation using user-level identity masking
  • 04DCAT architecture that decouples context and crossing transformer operations
  • 05Maximizing serving efficiency with custom Triton kernels and KV caching

RECOMMENDATION

Essential reading for recommendation system engineers handling large user sequences and system architects focused on cost-efficient ML scaling.

The Problem

As recommendation models scale, massive user sequence data is redundantly processed and stored for thousands of items per request, causing exponential infrastructure pressure and costs.

The Solution

The team implemented request-level deduplication across the ML lifecycle using request-sorted storage with Apache Iceberg, SyncBatchNorm, user-level masking for training, and a specialized DCAT architecture.

The Result

Achieved 10-50x storage compression, up to 4x training speedup, and a 7x increase in serving throughput, enabling a 100x model scale-up without proportional cost increases.

Trade-off

Request-sorting disrupts the IID assumption and increases the false negative rate in retrieval models, requiring additional architectural corrections and complexity.

03

Key Concepts

Concept · 01

Request-Level Deduplication

An optimization technique ensuring that shared context or user data within a single request is processed and stored only once instead of once per item.

  • Eliminates redundancy during data movement and transformation
  • Acts as a cross-cutting technique impacting storage, training, and serving
Concept · 02

SyncBatchNorm

A normalization layer that synchronizes statistics across all distributed devices to ensure a more representative global mean and variance.

  • Mitigates bias caused by request-sorted batches dominated by single users
  • Fully recovers the performance gap caused by non-IID training data
Concept · 03

DCAT (Deduplicated Cross-Attention Transformer)

A transformer variant that separates user history encoding from candidate item cross-attention to enable deduplicated computation.

  • Avoids redundant re-computation of user sequences for every scored item
  • Drives a 7x throughput gain in production serving environments