arXiv Computer Vision

Guiding Image-to-3D Generation with Test-Time Partial Observations

arXiv Machine Learning
Sep 4

Sparse auto-regressive modeling for scene generation from multi-view images

The paper introduces SPAR3S, a sparse voxel‑aligned 3D latent generative model that completes 3D scenes from sparse, unconstrained multi‑view images. It learns a compact voxel‑aligned latent space using photometric supervision via differentiable 3D Gaussian Splatting, and employs a masked autoregressive transformer to predict missing voxel occupancy and latent tokens. Experiments on synthetic indoor scenes and RealEstate10k show that SPAR3S achieves higher novel‑view quality than prior methods and generalizes to real‑world data.

By Thomas Lucas, Maxime Pietrantoni, Philippe Weinzaepfel, Wonjune Cho, Bardienus Pieter Duisterhof, Vincent Leroy, Jerome Revaud
arXiv Computer Vision
Sep 3

RoGe: Novel View Synthesis via End-to-End Implicit Reconstruction and Generation

RoGe is a new end‑to‑end framework for novel view synthesis that jointly learns an implicit 3D scene representation and a video diffusion model. It eliminates the need for explicit 3D intermediates by querying the implicit scene with camera rays to produce geometric features that condition the diffusion model. Experiments on DL3DV show that RoGe surpasses reconstruction‑based, generation‑based, and hybrid baselines in image quality and temporal consistency, and ablations confirm the benefits of ray‑queried features and joint training.

By Xiaolei Lang, Ze Kang, Zehao Huang, Naiyan Wang
Hugging Face Trending Papers
Sep 3

Sparse auto-regressive modeling for scene generation from multi-view images

The paper introduces SPAR3S, a sparse voxel‑aligned 3D latent generative model that completes 3D scenes from sparse, unconstrained multi‑view images. By representing only occupied voxels in a compact latent space and training a masked autoregressive transformer with photometric supervision via differentiable 3D Gaussian Splatting, the method predicts missing latent tokens and spatial support, enabling efficient and spatially consistent generation of unseen regions. Experiments on synthetic indoor scenes and RealEstate10k demonstrate higher novel‑view quality and real‑world applicability compared to prior work.

arXiv Computer Vision
Aug 24

RecGen3D: Reconstruction-Guided 3D Generation in a Shared Canonical Space

RecGen3D is a framework that merges feed‑forward reconstruction and diffusion‑based generation to address the trade‑off between reconstruction fidelity and generative plausibility in sparse‑view 3D modeling. By aligning both models in a shared canonical space and using decoupled cooperative learning, the system stabilizes training and allows the reconstruction module to supply canonical geometric anchors while the diffusion generator refines and completes the structure. Experiments show that RecGen3D outperforms existing methods in producing complete and consistent 3D models from sparse observations.

By Zhisheng Huang, Jiahao Chen, Cheng Lin, Chenyu Hu, Hanzhuo Huang, Zhengming Yu, Mengfei Li, Yuheng Liu, Zekai Gu, Zibo Zhao, Yuan Liu, Xin Li, Wenping Wang
arXiv Computer Vision
Sep 7

WorldSculpt: Generating Compositional Worlds from Grounded Videos

WorldSculpt presents a method for generating compositional 3D representations of cluttered scenes with hundreds of objects by adapting a single-object 3D generative prior to multi-view observations. The approach, built on Pixal3D with a multi-view conditioning pathway, can generalize to highly occluded scenes without scene-level training. The authors also introduce the UE-MeshyScene benchmark and demonstrate that their method outperforms prior approaches across various evaluation settings, including converting existing 3DGS worlds into compositional mesh scenes.

By Muyao Niu, Jixuan He, Ruihan Yu, Lian Fu, Yonghao Yu, Zheng-Hui Huang, Yifan Zhan, Fengbo Lan, Yongtao Ge, Yinqiang Zheng, Kaipeng Zhang, Zhixiang Wang
arXiv AI
Jun 4

SAM 3D: 3Dfy Anything in Images

arXiv:2511. 16624v2 Announce Type: replace-cross Abstract: We present SAM 3D, a generative model for visually grounded 3D object reconstruction, predicting geometry, texture, and layout from a single image.

By SAM 3D Team, Xingyu Chen, Fu-Jen Chu, Pierre Gleize, Kevin J Liang, Alexander Sax, Hao Tang, Weiyao Wang, Michelle Guo, Thibaut Hardin, Xiang Li, Aohan Lin, Jiawei Liu, Ziqi Ma, Anushka Sagar, Bowen Song, Xiaodong Wang, Jianing Yang, Bowen Zhang, Piotr Doll\'ar, Georgia Gkioxari, Matt Feiszli, Jitendra Malik
arXiv Computer Vision
Aug 28

SpatialCrafter: Single Image World Modeling with Generative 3D Proxies

SpatialCrafter introduces a two‑stage framework for single‑image world modeling that first generates a global 3D proxy using a Point‑anchored Sparse Structure Flow module, then refines appearance with a Generative Deferred Refiner built on a video diffusion model. The method incorporates Parallel Geometry Injection and Proxy‑Aware Corruption training to integrate the proxy without disrupting the pretrained generative manifold, and it is evaluated on a newly constructed dataset of 115K scenes. Experiments demonstrate that SpatialCrafter outperforms existing approaches, reducing long‑term drift and maintaining consistency under rapid camera motion and extreme viewpoints.

By Chuan Fang, Lingteng Qiu, Yixun Liang, Rui Chen, Kunming Luo, Zhaohua Zheng, Tongyuan Bai, Feipeng Tian, Zilong Dong, Zihan Zhou, Ping Tan
arXiv Computer Vision
Aug 31

Video Generative Models as Geometry Learner

The paper introduces GeoNeXt, a framework that repurposes pretrained video generative models for geometry estimation by framing it as a next‑frame prediction task. Unlike prior methods that either train separate depth/normal models or fine‑tune image diffusion backbones, GeoNeXt jointly models images and geometric targets, leveraging the structured knowledge of video models for more data‑efficient learning. Experiments show zero‑shot monocular depth and surface normal estimation that outperforms existing generative approaches and rivals discriminative state‑of‑the‑art methods while using far less training data.

By Haosen Yang, Jifei Song, Zhensong Zhang, Xiatian Zhu, Jiankang Deng