arXiv Computer Vision

GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space

arXiv Computer Vision
Sep 22

GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation

The paper introduces the Geometry‑Native Autoencoder (GAE), a compact latent space that can be decoded into appearance, depth, camera parameters, and point maps, enabling 3D‑consistent world generation. By reparameterizing a geometry foundation model’s features, GAE replaces traditional appearance‑centric latents and improves visual quality and 3D coherence, achieving significant reductions in FVD and camera‑trajectory error on benchmark datasets. The work demonstrates that a geometry‑native latent space can serve as a shared interface between perception and generation models.

By Jiahao Lu, Minghao Yin, Wenbo Hu, Hengyu Liu, Wang Zhao, Sai-Kit Yeung, Ying Shan, Yuan Liu
arXiv Computer Vision
Aug 28

SpatialCrafter: Single Image World Modeling with Generative 3D Proxies

SpatialCrafter introduces a two‑stage framework for single‑image world modeling that first generates a global 3D proxy using a Point‑anchored Sparse Structure Flow module, then refines appearance with a Generative Deferred Refiner built on a video diffusion model. The method incorporates Parallel Geometry Injection and Proxy‑Aware Corruption training to integrate the proxy without disrupting the pretrained generative manifold, and it is evaluated on a newly constructed dataset of 115K scenes. Experiments demonstrate that SpatialCrafter outperforms existing approaches, reducing long‑term drift and maintaining consistency under rapid camera motion and extreme viewpoints.

By Chuan Fang, Lingteng Qiu, Yixun Liang, Rui Chen, Kunming Luo, Zhaohua Zheng, Tongyuan Bai, Feipeng Tian, Zilong Dong, Zihan Zhou, Ping Tan
arXiv Computer Vision
Sep 3

RoGe: Novel View Synthesis via End-to-End Implicit Reconstruction and Generation

RoGe is a new end‑to‑end framework for novel view synthesis that jointly learns an implicit 3D scene representation and a video diffusion model. It eliminates the need for explicit 3D intermediates by querying the implicit scene with camera rays to produce geometric features that condition the diffusion model. Experiments on DL3DV show that RoGe surpasses reconstruction‑based, generation‑based, and hybrid baselines in image quality and temporal consistency, and ablations confirm the benefits of ray‑queried features and joint training.

By Xiaolei Lang, Ze Kang, Zehao Huang, Naiyan Wang
arXiv Computer Vision
Aug 28

Latent Spatial Memory for Video World Models

The paper introduces latent spatial memory, a 3D cache that stores scene information directly in diffusion latent space, eliminating the need for pixel-space reconstruction. It presents Mirage, a framework that lifts latent tokens into 3D using depth-guided back‑projection and queries the memory via latent‑space warping, achieving significant speed and memory gains. Experiments demonstrate up to 10.57× faster video generation, a 55× reduction in memory usage, and state‑of‑the‑art performance on WorldScore and strong reconstruction on RealEstate10K.

By Weijie Wang, Haoyu Zhao, Yifan Yang, Feng Chen, Zeyu Zhang, Yefei He, Zicheng Duan, Donny Y. Chen, Yuqing Yang, Bohan Zhuang
arXiv Computer Vision
Nov 9

Latent Riemannian Flow Matching for Geometry-Grounded 3D Foundation Models

The paper introduces Latent Riemannian Flow Matching, a method that performs flow matching directly in the latent space of geometric foundation models like VGGT. By respecting the product of zero‑mean hyperspheres imposed by LayerNorm, the approach avoids the pitfalls of treating the latent space as unconstrained Euclidean. Experiments on RealEstate10K, ScanNet++, and ETH3D show that this technique outperforms recent scene‑generation baselines in both novel‑view appearance and aggregated 3D geometry, and it transfers to other pretrained 3D backbones such as Depth Anything 3.

By Lisa Weijler, Irene Ballester, Guofeng Mei, Tolga Birdal, Pedro Hermosilla
arXiv Computer Vision
Oct 21

Latent Bridge: From Video Latents to 3D Geometry via Feature Alignment

Latent Bridge connects video generative latent spaces with geometric feature spaces, enabling direct prediction of 3D geometry—cameras, depth, and point clouds—without an intermediate RGB decoding step. The method trains representation‑specific bridges for Wan2.1 and V‑RAE via feature distillation on RGB videos while keeping the video encoder and geometry model frozen, requiring no ground‑truth geometry. At inference, the same video latents support both RGB decoding and geometry prediction, and the trained bridge can be reused across image‑to‑video and text‑to‑video generation, achieving competitive reconstruction quality on Co3Dv2 and WildRGB‑D with efficient end‑to‑end generation.

By Shuai Zhang, Zhanpeng Ouyang, Zhengqing Chen, Ruohong Mei, Shaoyu Chen, Weiqiang Ren, Hui Sun, Qian Zhang, Wenyu Liu, Xinggang Wang
arXiv Computer Vision
Nov 2

PRISM: Feed-Forward Single-Image 3D Reconstruction via Geometric Warp-Residual Modeling

PRISM is a feed‑forward single‑image 3D reconstruction framework that splits the task into a parameter‑free geometric warp and a learned residual correction, eliminating the need for iterative diffusion sampling. It uses a two‑stage training strategy—latent supervised distillation for geometric generalization and perceptual fine‑tuning for appearance quality—to generalize from synthetic data. Experiments on three benchmarks show PRISM matches diffusion‑based methods in quality while cutting inference time to just 36 seconds per scene.

By Zhijie Zheng, Xinhao Xiang, Jiawei Zhang
arXiv AI
Jun 9

VideoGPA: Distilling Geometry Priors for 3D-Consistent Video Generation

arXiv:2601. 23286v4 Announce Type: replace-cross Abstract: While recent video diffusion models (VDMs) produce visually impressive results, they fundamentally struggle to maintain 3D structural consistency, often resulting in object deformation or spatial drift.

By Hongyang Du, Junjie Ye, Xiaoyan Cong, Runhao Li, Jingcheng Ni, Aman Agarwal, Zeqi Zhou, Zekun Li, Randall Balestriero, Yue Wang
arXiv Computer Vision
Aug 31

Video Generative Models as Geometry Learner

The paper introduces GeoNeXt, a framework that repurposes pretrained video generative models for geometry estimation by framing it as a next‑frame prediction task. Unlike prior methods that either train separate depth/normal models or fine‑tune image diffusion backbones, GeoNeXt jointly models images and geometric targets, leveraging the structured knowledge of video models for more data‑efficient learning. Experiments show zero‑shot monocular depth and surface normal estimation that outperforms existing generative approaches and rivals discriminative state‑of‑the‑art methods while using far less training data.

By Haosen Yang, Jifei Song, Zhensong Zhang, Xiatian Zhu, Jiankang Deng