arXiv Computer Vision

Learning from Next-Frame Prediction: Autoregressive Video Modeling Encodes Effective Representations

arXiv Computer Vision
Oct 23

Learning from Next-Frame Prediction: Towards Unified and Scalable Visual Representation Learning

The paper introduces VANE, a self‑supervised visual pretraining framework that learns from next‑frame prediction in embedding space. It uses a causal encoder to summarize past observations, a state predictor to infer the representation of the entire next frame, and a lightweight generative decoder to reconstruct fine‑grained spatial detail. By applying the same objective to both videos and still images through a local‑to‑global crop sequence, VANE achieves state‑of‑the‑art performance on anticipation, recognition, and dense perception benchmarks, notably outperforming V‑JEPA on EPIC‑KITCHENS‑100 action anticipation.

By Yifeng Yao, Jinghan Li, Zhihao Zhang, Yajiao Xiong, Yang Jin, Kun Xu, Huishuai Zhang, Dongyan Zhao
arXiv Computer Vision
Oct 22

Looking Ahead: Modeling Future Evolution in Autoregressive Video Generation

The paper introduces Future Chunk Prediction (FCP), an auxiliary objective for autoregressive video diffusion models that predicts temporally distant chunks without seeing the intermediate frames. By jointly optimizing FCP with teacher‑forced denoising, the model learns to capture visual evolution rather than relying on local continuity, achieving better physical plausibility while using less training compute. Visualizations indicate that FCP encourages a more structured representation space, potentially benefiting future autoregressive world‑model development.

By Ruopeng Gao, Yukun Huang, Jiarui Hu, Shuai Wang, Wenbo Hu, Ying Shan, Limin Wang
Hugging Face Trending Papers
Aug 13

V-RAE: Rethinking Video Latent Spaces for Generation

Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization.

arXiv AI
Jul 13

Video Generation Models are General-Purpose Vision Learners

arXiv:2607. 09024v1 Announce Type: cross Abstract: Driven by next-token prediction, NLP shifted from task-specific models into powerful generalist foundation models.

By Letian Wang, Chuhan Zhang, Rishabh Kabra, Jasper Uijlings, Steven Waslander, Andrew Zisserman, Joao Carreira, Kaiming He, Misha Andriluka, Eduard Gabriel Bazavan, Andrei Zanfir, Cristian Sminchisescu
arXiv Machine Learning
Sep 24

Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders

The paper introduces LLMAE, a technique that transforms a pretrained decoder-only language model into a continuous text autoencoder by inserting a fixed-length latent bottleneck into its internal activations. Using a 270M Gemma 3 model with structured attention masks, LoRA adaptation, and KL regularization, LLMAE achieves near-perfect reconstruction of text sequences up to 1024 tokens. The authors further show that the resulting latent representation can be leveraged to train a latent text diffusion model for detailed image captioning, demonstrating downstream utility.

By Arkanath Pathak, Unnat Jain, Alexander C. Berg
arXiv AI
Oct 26

Representation Learning via Composite Subspace Flow Matching

The paper introduces Composite Subspace Flow Matching (CSFM), a representation learning method that minimizes variance by inverting a deterministic degradation curriculum across spectral, color, and illumination subspaces. CSFM uses hard temporal style truncation to constrain an encoder, encouraging it to ignore transient style artifacts while preserving persistent content. Experiments on CIFAR-10, CIFAR-100, and STL-10 demonstrate that CSFM achieves competitive discriminative transferability and generative synthesis, outperforming standard optimal transport flows in linear probing and few‑shot classification while maintaining a smooth latent space.

By Jos\'e Afonso, M\'ario Figueiredo