KOEN
Read Original
#AI

Toward More Controllable AI Video Editing: An Early Research Exploration at Netflix

Toward More Controllable AI Video Editing: An Early Research Exploration at Netflix
01

Summary

Netflix's Vision for Controllable AI Video Editing: Change Only What Matters

Introducing Vera for Non-destructive Layered Editing and VOID for Physics-aware Object Deletion

Netflix Research presents two groundbreaking models aimed at solving the challenge of controllability in AI video editing. By enabling artists to dictate precise changes while preserving source integrity (Vera) and ensuring physical laws are respected during object removal (VOID), these tools represent a major leap toward professional-grade AI workflows.

  • 01Vera: A non-destructive approach that generates edits as separate layers while keeping original pixels untouched
  • 02VOID: Goes beyond simple inpainting by simulating physical consequences, such as objects falling after a carrier is removed
  • 03Mixture-of-Transformers (MoT): Utilizes specialized DiTs for edit layers, alpha mattes, and composites to ensure high quality
  • 04Human studies with creative professionals confirm significant preference for Vera and VOID over current industry baselines
  • 05VLM-based Reasoning: Employs visual-language models to identify and guide regions affected by causal physical changes

RECOMMENDATION

Highly recommended for video editors and AI researchers seeking precise control over generative outputs. The layered generation approach is particularly relevant for integration into existing professional production pipelines.

The Problem

Existing generative video editing models often regenerate the entire video even for minor edits, leading to unintended changes in identity or background. Furthermore, they frequently fail to maintain physical continuity and interactions when objects are removed from a scene.

The Solution

Netflix developed 'Vera,' a layered diffusion model that generates only the necessary edits as separate layers, and 'VOID,' a video inpainting model that performs physically plausible reconstruction using VLM-based reasoning and quadmasks to guide the diffusion process.

The Result

Quantitative and human evaluations showed that Vera significantly outperformed baselines in content preservation. VOID was preferred in 64.8% of cases for complex scenarios requiring physical plausibility, demonstrating superior performance over existing open and closed-source models.

Trade-off

Current limitations include difficulty handling complex effects like smoke or lightning and domain gaps in unusual camera angles or extreme close-ups. There are also constraints regarding supported video resolution and maximum frame length.

03

Key Concepts

Concept · 01

Layered Video Diffusion

A technique where edits are generated as separate transparent layers and alpha mattes rather than regenerating the entire frame, allowing for perfect preservation of original content.

  • Used in Vera to maintain source identity and performance.
  • Enables seamless composition of new elements onto existing footage.
Concept · 02

Mixture-of-Transformers (MoT)

An architectural design that uses multiple specialized Transformer branches (DiTs) to handle different output distributions independently while allowing cross-layer interaction.

  • Vera uses three separate DiTs for the edit layer, alpha matte, and composite layer.
  • Improves data efficiency by allowing each branch to specialize in its specific output type.
Concept · 03

Physical Plausibility

The quality of a generated video where motions and interactions follow the laws of physics, making the result look realistic to the human eye.

  • VOID reconstructs scenes based on physical reasoning of what happens after an object is deleted.
  • Uses quadmasks to identify and stabilize regions affected by physical causalities.