arXiv Computer Vision

PROSE: Training-Free Egocentric Scene Registration with Vision-Language Models

arXiv AI
Jul 21

Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answering

arXiv:2510. 01483v3 Announce Type: replace-cross Abstract: Vision-language models (VLMs) demonstrate strong image-level scene understanding, but reasoning over long egocentric video remains costly: because VLMs maintain no persistent memory or explicit spatial representation, all sampled frames must be re-processed for every new query.

By Mohamad Al Mdfaa, Svetlana Lukina, Timur Akhtyamov, Arthur Nigmatzyanov, Dmitrii Nalberskii, Sergey Zagoruyko, Gonzalo Ferrer
arXiv Computer Vision
Sep 7

MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervision

MINT is a foundation model that directly predicts world-space two-hand trajectories from egocentric RGB video, jointly estimating camera motion, hand states, and hand presence in a single spatiotemporal representation. It uses an open-source labeling pipeline, EGOPIPELINE, to generate large-scale pseudo-labels for pretraining, followed by fine-tuning on a small set of high-quality joint annotations. The model outperforms existing multi-stage approaches in accuracy and speed, and generalizes zero‑shot to unseen egocentric datasets.

By Zijie Zhu, Weiren Cai, Yizhou Wang, Zhenjie Yang, Yide Liu, Jiahao Chen, Guanqi He
arXiv Computation and Language
Sep 30

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

arXiv:2609.38177v1 Announce Type: cross Abstract: Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs...

By Jaewoo Jung, Hyeonseo Yu, Honggyu An, Jisang Han, Mungyeom Kim, Minkyeong Jeon, Heeseong Shin, Wonjun Moon, Federico Tombari, Daniel Barath, Marc Pollefeys, Seungryong Kim, Sunghwan Hong
arXiv Computer Vision
Sep 21

A Scene Language Model for Open-Vocabulary Scene Mapping

The paper introduces SceneLM, a vision‑language model that maintains an open‑vocabulary 3D scene map using only a structured text list of objects as persistent memory. The model updates this textual map by adding, editing, and removing objects for each input image, learning the process through supervision tasks and an automatic annotation pipeline. Evaluations on language‑grounded retrieval and localization benchmarks show competitive performance with traditional mapping systems while producing a 6‑12× more compact representation, and the model can run online on an edge device such as a quadruped robot.

By Adam Lilja, Fabio H\"ubel, Siming He, Junsheng Fu, Claire Tomlin, Lars Hammarstrand, Jitendra Malik, Jonas Frey, Marco Pavone