KOEN
Read Original
#AI

MediaFM: The Multimodal AI Foundation for Media Understanding at Netflix

MediaFM: The Multimodal AI Foundation for Media Understanding at Netflix
01

Summary

How Netflix 'Watches' Everything: Introducing MediaFM, the Multimodal Core

Fusing Video, Audio, and Text into a Single Intelligent Backbone for Deep Media Understanding

Netflix unveils MediaFM, its first in-house foundation model that treats movies and shows as complex sequences of multimodal data. By learning the temporal and narrative relationships between shots, this Transformer-based model provides a sophisticated machine-readable understanding that powers everything from ad placement to trailer optimization.

  • 01Tri-modal architecture integrating Video (SeqCLIP), Audio (wav2vec2), and Timed Text
  • 02Self-supervised Masked Shot Modeling (MSM) objective for robust representation learning
  • 03Global context injection using title-level metadata via a dedicated Transformer token
  • 04Significant performance leap over generic commercial multimodal APIs
  • 05Proven insight that 'Context is King'—contextualized shots outperform isolated clips

RECOMMENDATION

Essential reading for ML engineers building media understanding systems or working with long-form multimodal sequence modeling.

The Problem

Netflix required a scalable, machine-level understanding of its vast catalog, including subtle narrative dependencies and emotional arcs. As new content types like live events and podcasts are added, the ability to integrate visual, audio, and textual signals became critical for various internal and member-facing applications.

The Solution

They developed the Netflix Media Foundational Model (MediaFM), a tri-modal (audio, video, text) Transformer-based encoder. The model uses Masked Shot Modeling (MSM) for self-supervised pretraining on sequences of shots, leveraging title-level metadata via a special [GLOBAL] token for context.

The Result

MediaFM outperformed strong baselines, including internal SeqCLIP and external models like Google VertexAI and TwelveLabs, across tasks like ads relevancy and clip popularity. Ablation studies revealed that contextualization within a larger sequence is the primary driver of its performance gains.

Trade-off

The model relies on zero-padding for missing text data in shots without dialogue and prioritizes a modular design with frozen representations, which may trade off some end-to-end task optimization for architectural flexibility.

03

Key Concepts

Concept · 01

Masked Shot Modeling (MSM)

A self-supervised learning objective where the model predicts the original embeddings of randomly masked shots within a sequence.

  • Randomly masks 20% of input shot embeddings per sequence
  • Optimized by minimizing cosine distance between predicted and ground-truth embeddings
Concept · 02

Tri-modal Fusion

The process of combining visual, auditory, and textual data into a unified representation for holistic analysis.

  • Concatenates SeqCLIP, wav2vec2, and OpenAI embeddings into a 2304-dim vector
  • Handles missing text data through zero-padding to maintain consistent input shapes
Concept · 03

Contextualization

Enhancing the representation of a single shot by considering its surrounding shots and overall title-level information.

  • Utilizes Transformer self-attention to capture temporal dependencies across up to 512 shots
  • Ablation tasks confirmed contextualization is the most critical factor for narrative-heavy tasks