arXiv AI

STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration

The paper introduces STEP, a State‑Aware Task Estimator and Planner that uses multi‑modal large language models to explicitly estimate system states and predict state transitions during task planning. By forecasting future states alongside actions, STEP reduces hallucinated actions and improves task‑convergent planning. In a simulated robot assembly task, STEP outperforms the state‑of‑the‑art by 32.8% in action executability and 14.8% in final‑state error.

arXiv AI

Sep 25

Design and Evaluation of LLM Chaining-Based Task Planning for General Purpose Service Robots

The paper introduces an LLM chaining architecture for General Purpose Service Robots that splits instruction classification and action generation into two stages, cutting prompt length by about 45% and boosting planning consistency. Evaluation on 100 synthetic GPSR commands across three language models shows consistent improvements over single-prompt methods, with up to +37 percentage points gain on local models. Real‑robot trials on the Toyota HSR confirm that while planning success improves, execution-layer failures remain the main obstacle to full task completion.

By Lucas Da Mota Bruno, Jiahao Sim, Yoshinobu Hagiwara

llms robotics benchmarks

arXiv AI

Sep 1

Scale-Plan: Scalable Language-Enabled Task Planning for Heterogeneous Multi-Robot Teams

arXiv:2603.08814v2 Announce Type: replace-cross Abstract: Long-horizon task planning for heterogeneous multi-robot systems is essential for deploying collaborative teams in real-world environments; y...

By Piyush Gupta, Sangjae Bae, Jiachen Li, David Isele

llms agents robotics benchmarks safety

arXiv Computer Vision

Sep 30

CogWAM: Aligning Semantic Cognition with World Action Modeling via Event-Driven Interfaces

CogWAM is a cognition‑guided world‑action model that links task reasoning with action learning via a persistent Semantic State, which records completed task events and the active subtask. The model updates this state only on semantic transitions, enabling task‑level context to persist across multiple action chunks. During inference, CogWAM removes the future‑prediction branch and directly generates actions from observations and the maintained state, achieving state‑of‑the‑art performance on RoboDojo and BiCoord and demonstrating efficient closed‑loop dual‑arm manipulation in real‑world experiments.

By Sen Wang, Liu Liu, Xinjiang Wang, Zequn Chen, Haoyi Jiang, Taojun Ding, Tingyang Xiao, Zhizhong Su, Jie Wang, Sanping Zhou

robotics benchmarks

arXiv AI

Jul 20

A Systematic Study of Large Language Models for Task and Motion Planning With PDDLStream

arXiv:2510. 00182v2 Announce Type: replace-cross Abstract: While we know that large language models (LLMs) can solve some planning problems, we do not understand the extent of these capabilities for robotics.

By Jorge Mendez-Mendez

llms robotics

arXiv AI

Oct 1

X-Planner: Event-Structured Task Planning for Embodied Intelligence

arXiv:2609.25187v2 Announce Type: replace Abstract: Task planning bridges high-level instructions and executable behavior in long-horizon manipulation, yet modern Vision-Language-Action (VLA) systems...

By Howard Lu, Shalfun Li, Porter Pan, Cris, Lumen, Cyril, Eric Hu, Lily Li, Maeve Zhang, Rain Sun, Robert Wang, KZ Zheng, Viggo Chen, Tim Ding, Regsis Cheng, YJ Xiao, Kian, Hai Lin, Alan Song, Elise Ma, Gody Li, Victor Yao, Yohann Tang, Ingrid Yu, Jason He, James Wang, Ryan Yu, Ping Yang, Chris Pan, Vincent Chen, Roy Gan, Hao Wang, Qian Wang

llms robotics multimodal

arXiv AI

Jun 10

RoboGPT-R1: Enhancing Robot Task Planning with Reinforcement Learning

arXiv:2510. 14828v3 Announce Type: replace Abstract: Improving the reasoning capabilities of embodied agents is crucial for robots to complete complex human instructions in long-view manipulation tasks successfully.

By Jinrui Liu, Bingyan Nie, Boyu Li, Yaran Chen, Yuze Wang, Shunsen He, Haoran Li

llms agents reinforcement-learning robotics fine-tuning benchmarks

arXiv AI

Jul 24

Vision-Language-Policy Model for Dynamic Robot Task Planning

arXiv:2512. 19178v2 Announce Type: replace-cross Abstract: Bridging the gap between natural language commands and autonomous execution in unstructured environments remains an open challenge for robotics.

By Jin Wang, Kim Tien Ly, Jacques Cloete, Jin Jin, Nikos Tsagarakis, Ioannis Havoutis

llms agents robotics fine-tuning multimodal

arXiv AI

Oct 15

Towards Long-horizon Embodied Agents with Tool-Aligned Vision-Language-Action Models

arXiv:2605.13119v2 Announce Type: replace-cross Abstract: Vision-language-action (VLA) models are effective robot action executors, but they remain limited on long-horizon tasks due to the dual burde...

By Zixing Lei, Changxing Liu, Yichen Xiong, Minhao Xiong, Yuanzhuo Ding, Zhipeng Zhang, Weixin Li, Siheng Chen

llms agents robotics multimodal benchmarks

arXiv AI

Aug 3

Combining Large Language Models and Symbolic Reasoning for Multi-Robot Temporal Planning through Explainable Knowledge Bases

arXiv:2502. 19135v2 Announce Type: replace Abstract: We present PLANTOR, a framework for generating and executing multi-robot task plans from natural-language task descriptions through LLM-assisted knowledge-base construction.

By Enrico Saccon, Matteo Saveriano, Edoardo Lamon, Luigi Palopoli, Marco Roveri

llms robotics benchmarks

arXiv Machine Learning

Oct 16

Policy-Aligned Predictive Verification for Adaptive VLA Action-Chunk Execution

The paper introduces a policy‑aligned predictive verifier (PAV) that evaluates candidate action prefixes from vision‑language‑action (VLA) policies to determine how far to execute before re‑planning. PAV predicts the consequences of prefixes and checks their compatibility with successful task completion, using trajectory‑level survival supervision and a localization prior to learn local judgments. Experiments on LIBERO, RoboTwin 2.0, and real‑world manipulation demonstrate that PAV improves the trade‑off between success rate and query cost compared to existing adaptive methods.

By Minkyu Kim, Junhoo Lee, Seungyeon Kim, Suhyun Jeon, Baekseung Kim, Donghoon Han, Nojun Kwak

robotics multimodal

arXiv Computer Vision

Oct 1

The Planning Limits of Latent World Models

arXiv:2609.39235v1 Announce Type: cross Abstract: World models offer a promising way to help robots understand how the physical world evolves and plan complex behaviours through imagination. Yet exis...

By Ali Alrasheed, Basim Azam, Naveed Akhtar

robotics multimodal

arXiv Computer Vision

Aug 27

Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization

Zero-WAM introduces a causal video-action model that enables robots to perform unseen manipulation tasks by following in-context human video guidance. The authors create HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks, and propose an in-context future chunk prediction objective to prevent shortcut learning. In simulation, Zero-WAM attains a 47.0% success rate on seven unseen tasks, outperforming the best video-action baseline by 29.5 percentage points, and demonstrates real‑world generalization to complex, long‑horizon, and fine‑grained tasks.

By Jiaming Zhou, Qihang Zhang, Gangwei Xu, Cunxin Fan, Yujie Zhao, Ruilin Wang, Yiming Luo, Shuai Yang, Xing Zhu, Yujun Shen, Junwei Liang, Yinghao Xu

llms robotics

Related stories