Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

The paper introduces Factor-Aware Predictive Learning (FAPL) to improve cinematic video generation by effectively leveraging multimodal large language model (MLLM) latents. FAPL trains a Video Diffusion Transformer (DiT) in three stages: grounding the DiT with frozen MLLM latents, enforcing factor-aware control through factor-span replacement, and optimizing temporal inference with limited-input representations while preserving full-observation reconstruction. Experiments show that FAPL outperforms baselines on sparse-to-video, factor-aware editing, and multi-shot generation tasks, demonstrating enhanced MLLM feature use, temporal control, and factor coordination.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

• llms
• diffusion
• multimodal