arXiv Machine Learning By Amani Kiruga, Zhiyi Li, Ruojin Cai, Hansen Lillemark, Xiaoshan Wu, Ayush Tewari, Yilun Du

UGG: Unified Generative Geometry Model

Read the original on arXiv Machine Learning →

UGG is a unified generative geometry model that jointly learns RGB video and geometric representations such as depth and camera pose. By assigning different noise levels across modalities and time, it treats visual generation and geometric prediction as conditional completions within a single generative framework. This approach allows learning from heterogeneous datasets, supports inference-time search over multiple scene hypotheses, and outperforms non-unified baselines in tasks like camera-controlled video generation, depth estimation, and camera pose prediction, even with sparse observations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computer Vision
Aug 31

Video Generative Models as Geometry Learner

The paper introduces GeoNeXt, a framework that repurposes pretrained video generative models for geometry estimation by framing it as a next‑frame prediction task. Unlike prior methods that either train separate depth/normal models or fine‑tune image diffusion backbones, GeoNeXt jointly models images and geometric targets, leveraging the structured knowledge of video models for more data‑efficient learning. Experiments show zero‑shot monocular depth and surface normal estimation that outperforms existing generative approaches and rivals discriminative state‑of‑the‑art methods while using far less training data.

By Haosen Yang, Jifei Song, Zhensong Zhang, Xiatian Zhu, Jiankang Deng
arXiv Computer Vision
Oct 21

CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction

CARI4D is a category‑agnostic method that reconstructs spatially and temporally consistent 4D human‑object interactions from monocular RGB videos at metric scale. It introduces a pose hypothesis selection algorithm that integrates predictions from foundation models, refines them via a learned render‑and‑compare paradigm, and reasons about contacts to satisfy physical constraints. Experiments demonstrate a 38% improvement on in‑distribution data and 36% on unseen data compared to prior work, and the model generalizes to unseen categories, enabling zero‑shot application to internet videos.

By Xianghui Xie, Bowen Wen, Yan Chang, Hesam Rabeti, Jiefeng Li, Ye Yuan, Gerard Pons-Moll, Stan Birchfield
arXiv Computer Vision
Sep 3

RoGe: Novel View Synthesis via End-to-End Implicit Reconstruction and Generation

RoGe is a new end‑to‑end framework for novel view synthesis that jointly learns an implicit 3D scene representation and a video diffusion model. It eliminates the need for explicit 3D intermediates by querying the implicit scene with camera rays to produce geometric features that condition the diffusion model. Experiments on DL3DV show that RoGe surpasses reconstruction‑based, generation‑based, and hybrid baselines in image quality and temporal consistency, and ablations confirm the benefits of ray‑queried features and joint training.

By Xiaolei Lang, Ze Kang, Zehao Huang, Naiyan Wang