arXiv Computer Vision

How Representational Priors Shape Generalization in World Action Models

arXiv Computer Vision
Aug 31

Video Generative Models as Geometry Learner

The paper introduces GeoNeXt, a framework that repurposes pretrained video generative models for geometry estimation by framing it as a next‑frame prediction task. Unlike prior methods that either train separate depth/normal models or fine‑tune image diffusion backbones, GeoNeXt jointly models images and geometric targets, leveraging the structured knowledge of video models for more data‑efficient learning. Experiments show zero‑shot monocular depth and surface normal estimation that outperforms existing generative approaches and rivals discriminative state‑of‑the‑art methods while using far less training data.

By Haosen Yang, Jifei Song, Zhensong Zhang, Xiatian Zhu, Jiankang Deng
arXiv Machine Learning
4d ago

LocaLeJEPA: Maximum Entropy Representations under Geometric Priors

LocaLeJEPA extends Joint-Embedding Predictive Architectures (JEPA) by introducing a manifold-theoretic reformulation that aligns representation learning with non‑Euclidean geometric priors. It incorporates a locally Euclidean distribution‑matching regularizer, LocaLeReg, to stabilize training and scale to large models. Experiments on ImageNet‑1K show that LocaLeJEPA outperforms the original LeJEPA across both hyperbolic and Euclidean backbones, improving linear and k‑NN classification as well as downstream tasks.

By Kartik Tandon, Alessio Borgi, Pavel Snopov, In\'es Garc\'ia-Redondo, Davide D'Ascenzo, Sebastiano Cultrera di Montesano, Fabrizio Silvestri, Pietro Li\`o, Francesca Dominici, Randall Balestriero, Claudio Battiloro
arXiv AI
Oct 2

Geometric Similarity in VLM Low-Level Vision Representations

The paper introduces GeoSim, a four‑level framework for analyzing how vision‑language models (VLMs) represent low‑level vision tasks. It evaluates hidden‑layer representations across 24 tasks and two VLM paradigms—autoregressive models and diffusion transformers—using global similarity, local geometry, sparse feature decomposition, and topological verification. The study uncovers the organizing principles of low‑level visual representations and highlights their limitations in cross‑task and cross‑model agreement, offering an interpretability lens for assessing latent transferability and diagnosing model‑specific issues.

By Shao-Jun Xia, Huixin Zhang, Zhen Lei, Anlan Sun, Yuner Zhang, Xiaoyang Chen