arXiv AI

GeoBridge: Evolving Transferable Decoder Topologies for Geometric Vision

arXiv Computer Vision
Oct 21

HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation

HERMES++ is a unified driving world model that simultaneously performs 3D scene understanding and future geometry prediction. It uses a BEV representation to merge multi‑view spatial data for LLM compatibility, introduces LLM‑enhanced world queries for knowledge transfer, and employs a Current‑to‑Future Link to condition future geometry on semantic context. A Joint Geometric Optimization strategy enforces structural integrity by combining explicit geometric constraints with implicit latent regularization, and extensive benchmark tests show that HERMES++ outperforms specialist methods in both point cloud prediction and scene understanding.

By Xin Zhou, Dingkang Liang, Xiwu Chen, Feiyang Tan, Dingyuan Zhang, Hengshuang Zhao, Xiang Bai
arXiv Machine Learning
Oct 9

One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts

The paper introduces reViT, a recurrent vision transformer that uses a single Transformer block repeatedly to achieve the accuracy of a full-depth encoder while keeping inference FLOPs comparable. It achieves depth‑specific transformations by expressing each block’s feed‑forward network as a convex combination of a small shared expert bank, with a continuous depth coordinate guiding the mixture. Experiments on ImageNet‑1k and DINOv2 distillation show that reViT matches or surpasses state‑of‑the‑art models, reduces stored parameters by about 70%, and supports elastic‑depth deployment from a single checkpoint.

By Adrian Bulat, Yassine Ouali, Georgios Tzimiropoulos
arXiv Computer Vision
Oct 5

GeoScaffold: Learning Compact Geometric Latents via Reconstruction for Efficient Vision-Language Navigation

GeoScaffold introduces a geometric supervision framework that embeds 3D geometry into vision‑language navigation policies during training, eliminating the need for depth sensors or 3D encoders at inference. The method learns a compact depth tokenizer from training trajectories, then fine‑tunes the policy with geometry query tokens that reconstruct depth, connectivity, and traversability, turning these states into compact geometric latents for action decoding. After training, the tokenizer, generators, and reconstruction heads are discarded, leaving a lightweight backbone that outperforms vision‑only navigators on continuous VLN benchmarks.

By Yixuan Jiang, Wentong Li, An Liu, Zihao Xin, Fulin Tang, Cong Leng, Yang Gao, Jian Cheng
arXiv AI
Aug 28

Successive Capacity Growth: Task-Complexity-Driven Width and Depth Expansion for Vision Transformer Encoders in JEPA World Models

The paper introduces Successive Capacity Growth (SCG), a method for adaptively expanding Vision Transformer encoders in Joint-Embedding Predictive Architectures (JEPAs). SCG starts with a minimal encoder and incrementally increases width or depth based on a task‑agnostic test‑and‑verify mechanism, while a Sketched Isotropic Gaussian Regularizer (SIGReg) keeps learned semantic dimensions independent. Experiments on multi‑object dynamics and 2D navigation tasks show that SCG achieves up to 20.3% better prediction loss than fixed small baselines and 23% better than fixed large models, with far greater parameter efficiency and no false‑positive expansions.

By Frederik Berenz
arXiv Computer Vision
Oct 21

Latent Bridge: From Video Latents to 3D Geometry via Feature Alignment

Latent Bridge connects video generative latent spaces with geometric feature spaces, enabling direct prediction of 3D geometry—cameras, depth, and point clouds—without an intermediate RGB decoding step. The method trains representation‑specific bridges for Wan2.1 and V‑RAE via feature distillation on RGB videos while keeping the video encoder and geometry model frozen, requiring no ground‑truth geometry. At inference, the same video latents support both RGB decoding and geometry prediction, and the trained bridge can be reused across image‑to‑video and text‑to‑video generation, achieving competitive reconstruction quality on Co3Dv2 and WildRGB‑D with efficient end‑to‑end generation.

By Shuai Zhang, Zhanpeng Ouyang, Zhengqing Chen, Ruohong Mei, Shaoyu Chen, Weiqiang Ren, Hui Sun, Qian Zhang, Wenyu Liu, Xinggang Wang