Robotics and embodied AI

Manipulation, locomotion, sim-to-real transfer and autonomous driving: learning systems that have to survive physics.

3,858 stories · RSS feed

arXiv AI
Sep 24

The Gaussian Is Enough: Flow-Matching Priors Do Not Help When Fine-Tuning Large Behavior Models

The paper investigates whether using non‑Gaussian priors improves fine‑tuning of large behavior models (LBMs) for robot imitation learning. Across more than 100,000 simulation rollouts and 1,250 hardware trials on diverse tasks, the authors find that non‑Gaussian priors do not yield better fine‑tuning performance than standard Gaussian priors, except possibly at very low data fractions. Diagnostic analyses reveal that encoder training dominates fine‑tuning outcomes, while prior choice has minimal impact.

By Chen Xu, Rishi Shah, Hadas Kress-Gazit, Haruki Nishimura, Masha Itkina
arXiv AI
Sep 24

Teach-to-Crash: A Closed-Loop Student-Teacher LLM Framework for Collision-Inducing Test Scenario Generation

Teach-to-Crash is a closed‑loop testing framework that uses a dual‑LLM architecture to generate collision‑inducing scenarios for autonomous driving systems. A high‑reasoning Teacher LLM controls the search when collision metrics stagnate, while a low‑reasoning Student LLM produces simulator‑executable scenarios in JSON. In a CARLA case study, Teach‑to‑Crash achieved the highest collision hit rate (90.79 %), the shortest mean time‑to‑collision (18.31 s), and superior diversity and avoidability metrics compared to other methods.

By Zaid Ghazal, Khouloud Gaaloul, Bruce Maxim
arXiv AI
Sep 24

Breaking Weather-Content Coupling: Type-Severity Guided Progressive Disentanglement for All-in-One Infrared Restoration

The paper introduces TSGPD-IR, a network that disentangles weather-induced artifacts from true thermal signals in infrared images. It uses weather semantics and regional degradation severity to generate adaptive prompts, estimate severity without manual labels, and select appropriate expert modules for restoration. This approach aims to reduce artifacts and preserve weak thermal details across varying weather conditions.

By Xinyao Wang, Lijun He, Zhihan Ren, Fan Li
arXiv AI
Sep 24

Passing: An Endless Journey through Reconstructed Spacetime with AI-Generated Sound

Passing is an interactive audiovisual installation that transforms a single continuous monorail-window recording into an endless journey by reconstructing it as a spatiotemporal volume and resampling its spatial and temporal structure along nonlinear trajectories. A camera-based viewer‑presence detection system influences transitions among rendered video sequences, and the resulting video stream is fed into SpecMaskFoley, a real‑time video‑to‑audio synthesis model that generates a synchronized soundscape. The work distributes creative agency among the artist, the AI model, and the audience, exploring how authorship and listening can be negotiated among human intention, machine inference, and audience interpretation.

By Akira Takahashi, Chihiro Nagashima, Zhi Zhong, Shusuke Takahashi, Yuki Mitsufuji
arXiv AI
Sep 24

Kairos: Grounded Forecasting of Presence and Directional Flow in 4D Scene Graphs

Kairos extends a hierarchical 3D scene graph to a 4D scene graph, storing for each voxel a directional mixture and a presence rate. Spectral predictors forecast both the probability of people being present and the full directional distribution of their motion at any future query time. The model supports conditional queries via pairwise flow dependence and provides calibrated credible intervals that tighten as observations accumulate.

By Iacopo Catalano, Julio A. Placed, Javier Civera, Jorge Pe\~na Queralta
arXiv AI
Sep 24

Controlling Collectives of AI Agents in Reasoning Space with Spatial Transformers

The paper introduces COMPASS, a decentralized architecture that uses spatial transformers to generate local feedback tokens for large collectives of AI agents in robotics. By aggregating multi‑hop messages, COMPASS enables scalable control of up to 1024 robots, achieving cohesive flocking formations and accurate execution of natural language commands. Experiments show that structured diversity in input commands improves performance and that learned feedback tokens outperform hand‑crafted raw state feedback.

By Frederic Vatnsdal, Roshan Gopal, Romina Garcia Camargo, Vijay Kumar, Alejandro Ribeiro
arXiv AI
Sep 24

AnchorReasoning: A Visual Grounding and Causal Reasoning Dataset in Long-Tail Autonomous Driving Scenarios

arXiv:2609. 28366v1 Announce Type: cross Abstract: Vision-language models (VLMs) offer a promising approach to long-tail autonomous driving, but existing driving datasets provide limited supervision for connecting decision-critical visual evidence with reasoning and planning.

By Zhipeng Bao, Wenjie Zhao, Tianle Zhu, Haohua Que, Chence Yang, Geng Yuan, Qianwen Li
arXiv AI
Sep 24

MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research

MobileGym is a browser-hosted, lightweight simulation platform designed for mobile GUI agent research. It offers verifiable outcome signals via deterministic, JSON-based state judging and supports scalable online reinforcement learning with hundreds of parallel instances on a single server. The platform includes a declarative task-definition framework, a structured AnswerSheet protocol, and a benchmark of 416 parameterized tasks across 28 apps, demonstrating strong sim-to-real transfer in a case study.

By Dingbang Wu, Rui Hao, Haiyang Wang, Shuzhe Wu, Han Xiao, Zhenghong Li, Bojiang Zhou, Zheng Ju, Zichen Liu, Lue Fan, Zhaoxiang Zhang
arXiv Computer Vision
Sep 24

Task-Prototype Guided Flow Matching for Few-Shot Generalization in Vision-Language Robot Manipulation

Task-Prototype Guided Flow Matching (TP-Flow) is a few‑shot manipulation framework that transforms support demonstrations into structured task‑prototype tokens to guide both the initial flow prior and the velocity field. It uses symmetric cross‑attention with learnable queries to extract phase‑level prototypes, parameterizes a task‑adaptive initial distribution, and injects prototype information through gated adaptive normalization. TP‑Flow is trained with an episodic support‑query objective and prototype contrastive regularization, achieving high success rates on the LEROBOT‑ARM‑SO101 platform while maintaining real‑time execution and low latency.

By Yizhao Wang, Guantao Zhang, Jingbo Wang
arXiv Machine Learning
Sep 24

EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management

EBRL is an asynchronous embodied reinforcement learning training system that overlaps rollout and training stages, pipelines simulation and generation across environment groups, and eliminates synchronization stalls. It employs a fine‑grained resource manager that pools CPU cores and GPU streaming multiprocessors, dynamically adjusting resource quotas and batch sizes based on stage profiles and runtime feedback. Experiments on RLinf with four policies and four simulation benchmarks show EBRL improves rollout throughput by 1.30–3.47× and training convergence by 2.5× over state‑of‑the‑art embodied RL systems.

By Liang Mi, Weijun Wang, Bowen Gao, Tianze Yu, Zixu Hao, Han Xiao, Xin Ding, Mingzhe Huang, Xin He, Lu Shi, Hao Wu, Haipeng Dai, Guihai Chen, Yunxin Liu, Ting Cao
arXiv Machine Learning
Sep 24

Generalizable Robotic Insertion with World Models

The paper introduces a framework that uses world models to enable robotic insertion across diverse parts. By combining proprioceptive data with wrist‑mounted camera visuals, a single model is trained on up to 90 tasks, achieving 56% zero‑shot success on unseen objects versus 7% for a model‑free baseline. The approach scales with more training objects and can be fine‑tuned for improved data efficiency and performance.

By Nicklas Hansen, Iretiayo Akinola, Yijie Guo, Jie Xu, Bingjie Tang, Hao Su, Xiaolong Wang, Abhishek Gupta, Dieter Fox, Yashraj Narang
arXiv Machine Learning
Sep 24

ForgetMimic: Motion Unlearning for Reinforcement Learning Humanoid Control

ForgetMimic is a motion-level unlearning method for reinforcement learning-based humanoid control. It selectively degrades performance on a chosen subset of motions while preserving the policy’s effectiveness on the remaining motions. Experiments on Unitree G1 and H2 robots across 12 motions show that the method successfully removes memory of designated motions without affecting other behaviors.

By Xukun Luan, Zhongxiang Lei, Chen Gong, Shaowei Li, Yuanguo Bi, Jinyan Liu
arXiv Machine Learning
Sep 24

Optimization without Future Compromises? Decentralized Coordination via Collective and Reinforcement Learning

The paper introduces Hierarchical Reinforcement and Collective Learning (HRCL), a framework that combines multi‑agent reinforcement learning (MARL) with decentralized coordination. HRCL uses MARL at a high level to generate strategic guidance that limits the decision space for low‑level agents, enabling efficient short‑term coordination while considering long‑term effects. Experiments on synthetic, energy‑management, and drone‑swarm scenarios demonstrate faster convergence and significant reductions in system‑wide and individual costs compared to standalone MARL.

By Chuhao Qin, Evangelos Pournaras
arXiv Computer Vision
Sep 24

SatUnreal: A High-Precision Synthetic Dataset for Satellite Stereo Matching via Unreal Engine

SatUnreal is a synthetic dataset created with Unreal Engine that offers 10,000 high‑resolution (0.3 m GSD) satellite stereo pairs. It addresses key limitations of existing benchmarks by ensuring physical geometry simulation, spatio‑temporal consistency, topographic diversity, and mathematically precise occlusion masks via a two‑step line‑trace algorithm. Models trained solely on SatUnreal outperform those trained on real datasets when transferred to real‑world benchmarks such as US3D and WHU‑Stereo.

By Han-Gyeol Kim, JaeWan Park, Junmin Park, Darongsae Kwon
arXiv Computer Vision
Sep 24

VLMs Can Describe, But Not Measure: Object-Centric Scene Understanding for Robotic Manipulation

The paper introduces a modular perception framework that uses vision‑language models (VLMs) to annotate object‑level regions from a single RGB‑D observation, then grounds these annotations with depth data to build an object‑centric representation. Experiments on 151 tabletop scenes demonstrate that this decomposition maintains strong semantic performance while significantly improving localization and depth estimation compared to direct VLM inference. The resulting representation is integrated into a task‑planning system for robotic manipulation.

By Enrico Saccon, Tommaso Faraci, I\~{n}igo De La Ossa Zarzuelo, Luigi Palopoli, Marco Roveri, Matteo Saveriano
arXiv AI
Sep 24

Reinforcement Learning with Decomposed Subtasks

The paper introduces Reinforcement Learning with Decomposed Subtasks (RLDS), a method that splits trajectory rewards into per‑subtask shares before policy updates, replacing the scalar advantage used in Group Relative Policy Optimization (GRPO). RLDS employs Subtask‑Decomposed Advantage Estimation (SDAE) to compute group‑relative advantages and distribute credit to tokens based on subtask importance, focusing on steps where a reflection marks a subtask as consequential. Experiments on four benchmarks—FrozenLake, HotpotQA, ScienceWorld, and DeepResearch—show that RLDS improves performance on high‑heterogeneity tasks (ScienceWorld and FrozenLake) and is more compute‑efficient than scalar GRPO for long rollouts.

By Mattie Terzolo, Mikolaj Sacha, Ayan Sinha, Andrew Rabinovich
arXiv AI
Sep 24

Frozen Flows Forget: Diagnosing and Restoring Lost Motion in a Latent-flow World Model

The paper investigates why latent‑flow world models that use a frozen self‑supervised latent space lose the ability to manipulate motion. It shows that the pretrained flow does not move the manipulated object and that training with latent‑only losses only produces stillness or teleport‑like motion. The authors introduce Decode‑Augmented Rollout Training (DART), which keeps the representation frozen but retrains the flow using decode‑path supervision, restoring temporal motion structure and improving prediction quality, even closing much of the gap to an oracle‑informed reference. The study also notes that pixel error alone can favor frozen predictions.

By Xiwen Chen, Rigaudiere Z. Li, Zhiruo Zhou, Xiaojun Zhu, Houde Liu
arXiv AI
Sep 24

DreamAvoid: Critical-Phase Test-Time Dreaming to Avoid Failures in VLA Policies

DreamAvoid introduces a test‑time dreaming framework for Vision‑Language‑Action models to anticipate and avoid failures during critical manipulation phases. It uses a Dream Trigger to detect critical phases, samples candidate action chunks via an Action Proposer, and evaluates short‑horizon futures with a Dream Evaluator trained on success, failure, and boundary data. Experiments on real‑world and simulated tasks show DreamAvoid improves task success rates, achieving 72.5% success versus 48.8% for the base policy and 54.4% for GPC‑RANK.

By Xianzhe Fan, Yuxiang Lu, Shenyuan Gao, Xiaoyang Wu, Ruihua Han, Manling Li, Hengshuang Zhao
arXiv AI
Sep 24

UniShield: An Adaptive Multi-Agent Framework for Unified Forgery Image Detection and Localization

UniShield is a multi‑agent framework that unifies forgery image detection and localization across diverse domains such as image manipulation, document manipulation, DeepFake, and AI‑generated images. It combines a perception agent that analyzes image features to select appropriate detection models with a detection agent that integrates multiple expert detectors into a single system, producing interpretable reports. Experiments demonstrate that UniShield outperforms existing unified approaches and domain‑specific detectors, achieving state‑of‑the‑art performance and greater practicality, adaptiveness, and scalability.

By Qing Huang, Zhipei Xu, Xuanyu Zhang, Xiangyu Yu, Jian Zhang