arXiv AI

3DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints

The paper introduces 3DWay, a method that predicts 3D-consistent waypoints for robot manipulation by first generating multi‑view consistent 2D waypoints and then triangulating them. This approach addresses the 3D ambiguity inherent in 2D trajectory predictions and leverages pretrained vision‑language models to provide explicit 3D motion specifications. Experiments demonstrate that 3DWay improves 3D spatial grounding and vision‑language reasoning, enhancing generalization for robot manipulation tasks.

Hugging Face Trending Papers

Sep 8

3DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints

The paper introduces 3DWay, a method that predicts 3D consistent waypoints for robot manipulation using multi‑view images. By first generating 2D waypoints that are consistent across views and then triangulating them, the approach provides explicit 3D motion specifications while leveraging pretrained vision‑language models. Experiments demonstrate that 3DWay improves 3D spatial grounding and vision‑language reasoning, enhancing the generalization of robot manipulation policies.

llms robotics multimodal

arXiv AI

Jul 1

3D HAMSTER: Bridging Planning and Control in Hierarchical Vision Language Action Models through 3D Trajectory Guidance

arXiv:2606. 31329v1 Announce Type: cross Abstract: Hierarchical Vision-Language-Action (VLA) models decouple high-level planning from low-level control to improve generalization in robot manipulation.

By Dongyoon Hwang, Byungkun Lee, Dongjin Kim, Hyojin Jang, Hoiyeong Jin, Jueun Mun, Minho Park, Hojoon Lee, Hyunseung Kim, Jaegul Choo

llms robotics multimodal benchmarks

arXiv AI

Jul 7

Geometry-Aware Motion Latents for Learning Robust Manipulation Policies

arXiv:2607. 04714v1 Announce Type: cross Abstract: Learning motion latents for robotic manipulation heavily relies on extracting motion patterns from visual sequences, yet effective action abstractions require understanding three-dimensional geometric transformations.

By Yunchao Zhang, Yijia Weng, Ruizhe Liu, Ming Hu, Leonidas Guibas, Yanchao Yang

robotics benchmarks

arXiv Machine Learning

Jun 4

PointAction: 3D Points as Universal Action Representations for Robot Control

arXiv:2606. 03943v1 Announce Type: cross Abstract: Video-Action Models (VAMs) leverage the broad visual dynamics captured by pre-trained video diffusion models, offering a promising path toward generalizable robot manipulation.

By Mutian Tong, Han Jiang, Qiao Feng, Lingjie Liu, Jiatao Gu

diffusion robotics benchmarks

arXiv Computer Vision

Sep 2

Dyn-3D: Unveiling and Resolving Ego-Motion Ambiguity in Vision-Language Models

arXiv:2609.01059v1 Announce Type: new Abstract: As Vision-Language Models (VLMs) tackle dynamic 3D spatial reasoning, ego-motion perception becomes essential to resolve monocular scale ambiguity. How...

By Jiayu Ding, Zhuodong Liu, Lei Zhang, Manyu Xiong, Hongbo Jin, Haoran Tang, Hongbo Zhang, Changen Zhu, Wenbo Xing

llms rag multimodal benchmarks

arXiv Computer Vision

Oct 21

MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction

arXiv:2606.18558v2 Announce Type: replace Abstract: Motion forecasting is central to visual intelligence: agents must anticipate how objects will move in order to plan actions, reason about physical...

By Jianing Zhang, Chenhao Zheng, Yajun Yang, Max Argus, Rustin Soraki, Winson Han, Taira Anderson, Chun-Liang Li, Shuo Liu, Jiafei Duan, Zhongzheng Ren, Jieyu Zhang, Ranjay Krishna

agents robotics benchmarks

arXiv AI

Sep 21

FOCAL-VLA: Subtask-Guided Geometry Distillation and Implicit World Modeling for Vision-Language-Action Models

FOCAL‑VLA is a framework that improves vision‑language‑action models by combining subtask‑guided geometry distillation with implicit world modeling. It transfers geometric knowledge from VGGT to focus on subtask‑relevant image regions and uses Track4World features to capture future 3D evolution, guiding action generation without running these models at inference time. Experiments demonstrate that FOCAL‑VLA outperforms baselines on both simulation benchmarks and real‑world manipulation tasks.

By Zhiyuan Gao, Di Wen, Yanxiang Zhan, Mohammad Khoshnazar, Jeroen Sch\"afer, Kunyu Peng, Michael Beetz

llms robotics efficiency multimodal benchmarks

arXiv Computer Vision

Oct 21

Learning the 4D World through Point Trajectory Forecasting

arXiv:2610.28594v1 Announce Type: new Abstract: Reliable multi-view forecasting requires videos from different cameras to describe the same evolving 3D scene. To this end, we introduce Video-Point Mo...

By Byungwoo Jeon, Sunghwan Hong, Dohyeon Kim, Jonghoon Lee, Jungwoo Park, Inhee Lee, Doosan Baek, Jinil Kim, Taehyun Lim, Hanbyul Joo, Marc Pollefeys, Saining Xie, Jinwoo Shin

diffusion robotics benchmarks

arXiv AI

Sep 30

Gondola: Grounded Vision Language Planning for Robotic Manipulation

arXiv:2506.11261v2 Announce Type: replace-cross Abstract: Vision-language-action (VLA) models have shown promising progress in robotic manipulation. However, directly mapping visual observations and...

By Shizhe Chen, Ricardo Garcia, Paul Pacaud, Cordelia Schmid

computer-vision robotics multimodal benchmarks safety

arXiv Computer Vision

Oct 27

GeoLift: Grounding Video Spatial Reasoning in Explicit 3D Maps

arXiv:2610.33558v1 Announce Type: new Abstract: Video spatial reasoning requires multimodal large language models (MLLMs) to understand how objects are arranged and related across viewpoints. While t...

By Xinglin Li, TingTing Long, Jingzhi Zhou, Jiajing Chen, Jian Yang, Jin Xie

llms multimodal

arXiv AI

Jul 14

Action Map Policy: Learning 3D Closed-loop Manipulation via Pixel Classification

arXiv:2607. 10706v1 Announce Type: cross Abstract: The action space poses a major challenge in robot learning, since it is often high-dimensional, can span long time horizons, and frequently admits multi-modal optimal solutions.

By Haojie Huang, Zhang Ye, Linfeng Zhao, Boce Hu, Mingxi Jia, Yu Qi, Ahmed Agha, Dian Wang, Robert Platt, Robin Walters

llms diffusion robotics multimodal

arXiv Computer Vision

Oct 22

GeoMem-VLA: Internalizing 3D Scene Understanding into Temporal Memory for Vision-Language-Action Models

arXiv:2610.29866v1 Announce Type: new Abstract: Geometric structure is essential for robotic manipulation and understanding the 3D world. Most VLA models rely on 2D representations of current observa...

By Qinfan Zhang, Yifan Luo, Haoyuan Shi, Guanlin Liu, Han Dong, Haozhe Shan, Yong Dai, Xiangcong Ren, Xiaozhu Ju

rag robotics multimodal benchmarks

Related stories