Robotics and embodied AI

Manipulation, locomotion, sim-to-real transfer and autonomous driving: learning systems that have to survive physics.

3,855 stories · RSS feed

arXiv Computer Vision
6d ago

PhysWAM: Physically Consistent World Action Model for Autonomous Driving

PhysWAM is a unified world-action model for autonomous driving that jointly denoises multiview video, metric depth, and ego motion using a flow‑matching transformer. It introduces Coupled Point Projection (CPP), a geometric constraint that aligns generated depth points with LiDAR data after applying the predicted SE(3) ego motion, thereby enforcing physical consistency. At inference, trajectory selection uses a simple label‑free consensus rule, and the model demonstrates strong planning performance, zero‑shot transfer to unseen environments, and accurate, temporally coherent depth and video predictions.

By Dhruv Parikh, Fengcheng Yu, Quankai Gao, Jiawei Yang, Junjie Ye, Maulik Bhatt, Thang Vu, Charles Ochoa, Rowan McAllister, Igor Vasiljevic, Rajgopal Kannan, Viktor Prasanna, Vitor Guizilini, Yue Wang
arXiv Computer Vision
6d ago

EndoPrior-GS: Dynamic Endoscopic Reconstruction with a Joint Texture Prior

EndoPrior-GS is a new pipeline for dynamic endoscopic reconstruction that combines frame-extracted vision heuristics with depth maps. It creates a joint texture prior using a tool-filtered tissue mask, a non-specular photometric filter, and anatomical salience, which guides primitive initialization and density control. Experiments on EndoNeRF and SCARED datasets show that EndoPrior-GS reduces Flow Error by 27.7% and 25.8% compared to representative methods while maintaining real-time rendering speed and competitive quality.

By Jiaqi Huang, Shidong Wang, Tong Xin, Kabita Adhikari
arXiv Computer Vision
6d ago

InsightMap: Structured Spatial Modeling for Embodied Multimodal Reasoning

InsightMap is a framework that uses top‑down maps as explicit spatial memory and action‑conditioned prediction targets for language‑guided navigation. It links historical views to labeled map locations and employs a shared multimodal backbone to jointly learn navigation action prediction and post‑action map generation, providing auxiliary training supervision. The approach supports a unified RGB‑D pipeline for navigation, visual question answering, situated reasoning, and 3D grounding, achieving state‑of‑the‑art results on R2R‑CE, RxR‑CE, ScanQA, SQA3D, ScanRefer, and outperforming baselines on the Unitree Go2 platform.

By Hongpei Zheng, Hujun Yin
arXiv Computation and Language
6d ago

ProgressCompass: Embodied Progress Reward Models Are Lost Without the Right Context

The paper introduces ProgressCompass, a framework that enhances Embodied Progress Reward Models (PRMs) by providing the necessary contextual information for accurate progress estimation in long manipulation tasks. It presents ContextProgress-Bench, a benchmark with 24 tasks that tests PRMs under three context-dependent scenarios—State Recall, Sequence Tracking, and Recurrence Disambiguation—showing that even history-aware PRMs struggle without proper context. By integrating a context-aware loop that leverages general-purpose vision‑language models, ProgressCompass reduces PRM progress error by up to 82% and improves rank agreement by 76%.

By Jianshu Zhang, Keliang Wu, Chengxuan Qian, Xiyuan Yang, Ce Zhang, Ariel Tian, Anbang Liu, Haoran Lu, Han Liu
arXiv Computer Vision
6d ago

Spatial-OPSD: Self-Improving Spatial Reasoning via Label-Free Self-Distillation

Spatial-OPSD is a label‑free self‑improvement framework for vision‑language models that leverages spatial priors such as depth, 3D relations, and camera geometry to provide dense token‑level supervision. During training, a privileged teacher uses these priors while the student learns from only the original visual‑language input, and a recursive round‑wise scheme allows repeated self‑improvement without moving the teacher. Across four VLM families, one round of Spatial‑OPSD improves the five‑benchmark average, and three rounds push a strong spatially specialized model to the open‑source frontier, achieving the highest average among open models and best results on three of five spatial reasoning benchmarks.

By Zhenyu Liu, Zhangquan Chen, Keyi Chen, Mingze Sun, Xiang An, Haodong Jing, Ruqi Huang
arXiv Computer Vision
6d ago

Track-and-Complete: Learning Humanoid Skills from a Single Failed Human Video

The paper introduces TRACC, a pipeline that learns humanoid skills from a single failed human video by first imitating the usable portion of the motion trajectory and then completing the task based on the inferred outcome. It treats the motion prefix before failure as prior knowledge and uses a task-completion reward to guide learning toward the intended goal without needing a successful demonstration. The method is evaluated on six failed tasks from the Oops! dataset, showing its effectiveness in learning from failures.

By Sarmad Idrees, Jongeun Choi
arXiv AI
6d ago

IronLLM: Forging Compact Edge-Native Language Models for Real-Time Embodied Intelligence

IronLLM-0.6B is a 654‑million‑parameter language model engineered for efficient on‑device inference, featuring a hybrid attention architecture, X‑MTP multi‑token prediction, and a lightweight verification head that yields a 1.48× decoding speedup. Trained on roughly 6.2 trillion tokens with a quality‑oriented pipeline and further refined via Multi‑Domain On‑Policy Distillation, the model adopts an Instruct‑Only design to meet low‑latency requirements. A lighter variant, IronLLM‑0.6B‑Light, replaces RMSNorm with Dynamic Tanh and streamlines costly components to enhance inference and quantization efficiency, offering a strong performance‑efficiency trade‑off for resource‑constrained deployment.

By Changdi Yang, Fengquan Jiao, Haochih Lin, Haoran Yang, Jing Xiao, Liangyu Huo, Suxin Lu, Tiance Chen, Wei Liu, Yinggan Xu, Yunxiang Lu, Zai Zheng, Zhirui Xie, Zhongyang Che, Ziyan Tang, Zuoxiang Zhao, Jian Yao
arXiv Computer Vision
6d ago

LongLive-Plug: Once-for-All Distillation for Video Generation

LongLive‑Plug is a once‑for‑all distillation framework that learns reusable LoRA adapters on a base video diffusion model, enabling training‑free, plug‑and‑play deployment to a wide range of downstream models. These adapters provide single‑pass classifier‑free guidance, few‑step sampling, and long‑context error correction for autoregressive generation, and remain effective even when downstream models add conditioning branches or expand output channels. The authors demonstrate that the approach works on 54 downstream models across three backbone families and eight task categories, including world modeling, robotics, editing, and multimodal generation.

By Shuai Yang, Luozhou Wang, Wei Huang, ZhiFei Chen, Bohan Zhang, Xiao Fu, Qianli Ma, Chen-Hsuan Lin, Weian Mao, Bryan Chu, Song Han, Yukang Chen
arXiv Computer Vision
6d ago

Speed in the Blind Spot: An Interpretability Analysis of Dynamic Perception in VLMs for Autonomous Driving

Vision‑Language Models (VLMs) used in autonomous driving are evaluated on their ability to understand velocity through three tasks: surrounding‑agent speed, current ego speed, and short‑horizon future ego‑speed. Experiments on nuScenes show that while current ego speed can be accessed internally, verbal outputs are poor and surrounding‑agent speed is weakly encoded; temporal cues and frame order are largely ignored. Specialized driving models (Alpamayo‑1.5) improve latent speed representations but still leave gaps between internal knowledge and verbal readout, indicating that good planning outputs do not guarantee reliable dynamic state recovery.

By Katharina Winter, Stefan Englmeier, Fabian B. Flohr
arXiv AI
6d ago

Illusory Truth or Mere Exposure? Model-Dependent Repetition Effects in LLM-Based Social Media Simulations

The study examines how four large language models (Gemma-3-4b-it, Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and GPT-5-nano) exhibit repetition effects in a social media simulation. Using a two‑phase within‑context design, researchers collected 336,000 ratings on truth, importance, sentiment, and interest for 100 statements across 10 feed variants and 3 replications. Results show distinct patterns: Gemma-3 displays a genuine Illusory Truth Effect, Qwen2.5 shows a mere exposure effect, GPT-5-nano shows no truth boost and mild skepticism, and Llama-3.1 shows a small truth boost with reduced evaluative dimensions.

By Azza Bouleimen, Nicol\`o Pagan, Anik\'o Hann\'ak
arXiv AI
6d ago

HorizonFlow: Variable-Length Planning for Offline Goal-Conditioned RL

HorizonFlow is a hierarchical planner for offline goal-conditioned reinforcement learning that treats the planning horizon as an output rather than a fixed input. It uses a subgoal route planner and an action-prefix controller, both employing insertion-based generation and flow matching, to jointly generate continuous plan content and its length. The method leverages the partially generated plan to guide token insertion and to steer generation toward shorter plans, achieving superior performance on Maze2D, Multi2D, and OGBench benchmarks.

By JunHyeok Oh, Zian Jang, Byung-Jun Lee
arXiv AI
6d ago

Generative Interactions: Weaving Multiparty Human Motion with Bilevel Latent Dynamics

The paper introduces BRAID, a hierarchical latent-variable model that generates multi-person human motion by explicitly modeling both group-level interaction dynamics and individual behavior conditioned on evolving group context. It treats social motion generation as a meta-transfer learning problem, learning shared interaction priors across datasets and adapting them to arbitrary context sets of observed people and joints. BRAID supports coherent generation under full, sparse, or partial observations and produces compact social-state vectors useful for downstream embodied-agent systems, with evaluations on social forecasting, tracking, in-filling, and response generation.

By Ojas Shirekar, Yash Surange, Agustinas Ju\v{c}as, Chirag Raman
arXiv AI
6d ago

Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning

arXiv:2609.36588v1 Announce Type: cross Abstract: We study reinforcement learning (RL) methods for cooperative multi-agent Vision-Language-Action (VLA) models. This problem is challenging because VLA...

By Ruixiao Xu, Wong Lik Hang Kenny, Zhiqian Liu, Jianing Guo, Hanxiao Li, Kejian Shi, Shuning Zhang, Pu Feng, Yongjia Ma, Yuqing Ma, Kai Chen, Qi Dou, Yaodong Yang, Xianglong Liu, Simin Li