KOEN
Read Original
#AI

MAPS: Netflix’s Multimodal Asset Personalization at Scale

MAPS: Netflix’s Multimodal Asset Personalization at Scale
01

Summary

Conquering Cold-Start: How Netflix Uses Multimodal AI to Personalize Artwork for New Titles

How Netflix integrates CLIP and MediaFM to deliver instant personalization for newly launched visual assets without historical interaction data

Netflix introduced MAPS to overcome the limitations of ID-based recommendation systems by embedding rich multimodal content directly into the asset representation. Leveraging CLIP for image-text mapping and MediaFM for fused video/audio/text representations, Netflix successfully solved the cold-start problem for newly launched media assets. This article walks through their unified model architecture, reward-based weighting, and cost-effective embedding validation proxy.

  • 01AI that Sees Beyond ID: Fusing 768-dimensional CLIP image embeddings directly into asset representations to understand visual themes and cast members immediately
  • 02Unified Model Consolidation: Consolidating five separate canvas-specific models into a single unified model, utilizing CLIP's resilience to aspect-ratio and crop changes
  • 03Reward-Based Weighting: Overcoming unbalanced impression volumes across formats by weighting training examples using long-term member satisfaction rewards
  • 04Hearing the Video via MediaFM: Enhancing video preview recommendations by leveraging Netflix's in-house foundation model that fuses visual, audio, and caption signals
  • 05Low-Cost Embedding Selection: Saving engineering costs by using a simple linear probe proxy task to evaluate candidate embeddings before conducting expensive end-to-end A/B tests

RECOMMENDATION

Highly recommended for recommendation system engineers in large-scale media or e-commerce platforms facing severe cold-start challenges. It provides excellent production-grade blueprints for model consolidation using unified embeddings and robust offline evaluation with Inverse Propensity Scoring (IPS).

The Problem

At the cold-start phase when new titles or UI formats launch, traditional ID-based recommendation models fail to provide personalized recommendations for new assets due to the lack of historical user interaction data.

The Solution

Netflix integrated pretrained CLIP image embeddings and MediaFM (its multimodal foundation model) to represent actual content, consolidated separate canvas models into a single unified model, and introduced reward-based weighting.

The Result

Consolidating five models into one and resolving cold-start issues yielded significant offline IPS performance lifts (including 5.69% on the short-panel canvas) and drove statistically significant lifts in core streaming and discovery metrics during online A/B testing.

Trade-off

To mitigate extreme data imbalance across canvases, complex reward-based weighting and Inverse Propensity Scoring (IPS) were required, which may introduce increased computational serving overhead for processing high-dimensional embeddings.

03

Key Concepts

Concept · 01

CLIP

A contrastive image-text pre-training model developed by OpenAI that maps visual and textual modalities into a single shared multi-dimensional vector space.

  • Netflix integrated its 768-dimensional embeddings into asset representations to immediately capture visual themes and talent information.
  • Used to calculate cosine similarity between user search queries and asset images for query-aware ranking on the Search Page.
Concept · 02

MediaFM

Netflix's proprietary in-house multimodal foundation model that fuses visual, audio, and timed-text signals from video shots into a unified embedding.

  • Employed to represent video previews by combining visual clips with dialogue and soundtrack cues to capture the overall tone.
  • Achieved superior results in online A/B tests over basic ID-based and visual-only baselines on the TV platform.
Concept · 03

Inverse Propensity Scoring (IPS)

A causal inference method that applies inverse probability weights to counteract bias in logged historical recommendation data.

  • Applied on exploration data where assets were served randomly to create an unbiased offline estimator for evaluating candidate models.
  • Served as the critical gatekeeper metric before deploying any candidate recommendation models to live A/B traffic.