KOEN
Read Original
#AI

Multi‑Ratio Video Resizing for Mobile Feed and Marketing

Multi‑Ratio Video Resizing for Mobile Feed and Marketing
01

Summary

How Netflix Automates Cinematic Video Reframing for Mobile Feeds

An intelligent multi-ratio cropping pipeline powered by active speaker detection and temporal optimization.

Netflix developed an automated multi-ratio video cropping pipeline to reframe 16:9 widescreen footage into 9:16, 4:5, and 1:1 formats for mobile feeds like 'Clips' and social campaigns. Beyond simple center cropping, the system employs AI to identify active speakers, track dynamic scene compositions, and apply smooth temporal camera movements. Integrated seamlessly into internal and external workflows, it speeds up asset production while preserving editorial creative control.

  • 01Automated shot splitting using neural network boundary detection and LipFlap-based active speaker detection
  • 02Focus region discovery aggregating face detection, active speaker status, and visual saliency maps
  • 03Temporal smoothing heuristics to eliminate shaky framing and deliver cinematic panning paths
  • 04Superior reframing quality compared to Adobe Premiere's Auto Reframe in internal evaluation sets
  • 05XML export support enabling direct integration with external timeline tools like Adobe Premiere

RECOMMENDATION

Recommended for media engineers and video production teams seeking to scale automated multi-format video editing across diverse mobile and marketing platforms without compromising cinematic quality.

The Problem

Manually reframing 16:9 video footage into mobile-friendly aspect ratios (9:16, 4:5, 1:1) for mobile feeds and social channels lacks scalability as catalog size and use cases grow.

The Solution

Built an automated three-stage pipeline utilizing shot detection with Active Speaker Detection, focus region discovery via face detection and saliency maps, and temporal path optimization, integrated into internal tools and Adobe Premiere XML workflows.

The Result

Enabled single-pass generation of 9:16, 4:5, and 1:1 ratio variants without additional computational overhead, outperforming Adobe Premiere's Auto Reframe in internal tests while significantly reducing production time.

Trade-off

Fully automated reframing cannot completely replace nuanced creative intent, requiring review UI controls and XML export capabilities for targeted manual fine-tuning.

03

Key Concepts

Concept · 01

Active Speaker Detection

A computer vision technique that identifies and tracks which individual in a multi-person video scene is currently speaking using visual and acoustic cues.

  • Combines an internal face detector with LipFlap to accurately pinpoint active speakers across live-action and animated titles
  • Splits continuous shots into logical sub-shots based on speaker turns to adjust reframing focus dynamically
Concept · 02

Focus Region Discovery

An algorithmic process that identifies and synthesizes the most salient area within a video frame that requires viewer attention.

  • Combines face detection outputs with visual saliency maps to determine primary character framing
  • Ranks multiple characters using active speaker status and relative object size to produce a continuous focus region
Concept · 03

Temporal Reframing

A filtering and optimization technique applied across consecutive frames to smooth out crop box paths and prevent camera shake.

  • Locks a static crop window when displacement is minimal to ensure visual stability
  • Computes smooth dynamic tracking paths when significant motion occurs to recreate intentional cinematic panning