arXiv AI

HINT-Plan: Human Intention-Aware Robot Task Planning in Context-Rich Environments using Vision Language Models

HINT-Plan is a new method that integrates human intention prediction into robot task planning by using Vision Language Models to infer high‑level human intentions from third‑person images. These intentions are converted into goal states and combined with hierarchical Scene Graphs to formulate joint task‑planning problems in context‑rich environments. In a photorealistic simulation, HINT-Plan achieved a 69.71% success rate, outperforming baselines by up to 35.29% and reducing functional conflicts.

arXiv AI

Aug 28

STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration

The paper introduces STEP, a State‑Aware Task Estimator and Planner that uses multi‑modal large language models to explicitly estimate system states and predict state transitions during task planning. By forecasting future states alongside actions, STEP reduces hallucinated actions and improves task‑convergent planning. In a simulated robot assembly task, STEP outperforms the state‑of‑the‑art by 32.8% in action executability and 14.8% in final‑state error.

By Maitrey Gramopadhye, Prakash Baskaran, Xiao Liu, Songpo Li, Soshi Iba

llms robotics multimodal benchmarks

arXiv AI

Aug 26

CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Driving

arXiv:2605.10426v3 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing reasoning mecha...

By Minqing Huang, Yujiao Xiang, Zihan Liang, Jiajie Huang, Jingqi Wang, Yuheng Zhou, Zhi Xu, Feiyang Tan, Hangning Zhou, Mu Yang, Gong Che

agents diffusion robotics multimodal benchmarks

arXiv AI

Jul 10

APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts

arXiv:2607. 08024v1 Announce Type: cross Abstract: Long-horizon robot planning requires jointly reasoning over semantic task structure and geometric feasibility.

By Emily Jin, Joy Hsu, Yiqing Xu, Weiyu Liu, Nick Haber, Jiajun Wu

robotics multimodal

arXiv AI

Jul 24

Vision-Language-Policy Model for Dynamic Robot Task Planning

arXiv:2512. 19178v2 Announce Type: replace-cross Abstract: Bridging the gap between natural language commands and autonomous execution in unstructured environments remains an open challenge for robotics.

By Jin Wang, Kim Tien Ly, Jacques Cloete, Jin Jin, Nikos Tsagarakis, Ioannis Havoutis

llms agents robotics fine-tuning multimodal

arXiv AI

Sep 16

Bridging Learned Visual Perception and Symbolic Belief-Space Planning

The paper introduces a new paradigm called VLM-as-probabilistic-grounder, which models the uncertainty of Vision‑Language Model (VLM) predicate groundings as a probability distribution over symbolic states. This probabilistic grounding allows belief‑space planning, producing more robust plans in partially observable settings. Experiments in simulated household robot environments demonstrate that this approach improves robustness and task success compared to deterministic grounding methods.

By Guy Azran, Michael Navat, Sarah Keren

llms agents robotics multimodal

arXiv AI

Jun 4

PerceptTwin: Semantic Scene Reconstruction for Iterative LLM Planning and Verification

arXiv:2606. 04226v1 Announce Type: cross Abstract: Simulation environments are useful for both robot policy learning and planning verification and validation.

By Charlie Gauthier, Sacha Morin, Liam Paull

llms robotics safety

arXiv AI

Sep 1

Scale-Plan: Scalable Language-Enabled Task Planning for Heterogeneous Multi-Robot Teams

arXiv:2603.08814v2 Announce Type: replace-cross Abstract: Long-horizon task planning for heterogeneous multi-robot systems is essential for deploying collaborative teams in real-world environments; y...

By Piyush Gupta, Sangjae Bae, Jiachen Li, David Isele

llms agents robotics benchmarks safety

arXiv AI

Jun 10

RoboGPT-R1: Enhancing Robot Task Planning with Reinforcement Learning

arXiv:2510. 14828v3 Announce Type: replace Abstract: Improving the reasoning capabilities of embodied agents is crucial for robots to complete complex human instructions in long-view manipulation tasks successfully.

By Jinrui Liu, Bingyan Nie, Boyu Li, Yaran Chen, Yuze Wang, Shunsen He, Haoran Li

llms agents reinforcement-learning robotics fine-tuning benchmarks

Hugging Face Trending Papers

Sep 15

Bridging Learned Visual Perception and Symbolic Belief-Space Planning

In partially observable settings, agents must act without full knowledge of the world state and rely on uncertain state-estimation pipelines. Obtaining grounded and verifiable symbolic plans under suc...

llms agents robotics multimodal

arXiv AI

Aug 11

GraphThink: Graph-Enhanced LLM Thinking for Long-Horizon Embodied Task Planning

arXiv:2608. 07905v1 Announce Type: new Abstract: Embodied agents using LLM-based planners often struggle with physical hallucinations, poor generalization to long-horizon tasks, and lack of environmental awareness.

By Chen Li, Sijie Cheng, Yuelin Zhang, Junxi Li, Maozhi Huang, Yang Liu, Wenbing Huang

llms agents robotics benchmarks safety

arXiv AI

Sep 2

ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models

arXiv:2505.13180v3 Announce Type: replace Abstract: Integrating Large Language Models with symbolic planners is a promising direction for obtaining verifiable and grounded plans, with recent works ex...

By Matteo Merler, Nicola Dainese, Minttu Alakuijala, Giovanni Bonetta, Pietro Ferrazzi, Yu Tian, Bernardo Magnini, Pekka Marttinen

llms robotics multimodal benchmarks

arXiv Machine Learning

Jun 2

Semantic-Geometric Task Representations for Bimanual Manipulation from Human Demonstrations to Robot Action Planning

arXiv:2601. 11460v2 Announce Type: replace-cross Abstract: Learning structured task representations from human demonstrations is essential for bimanual manipulation, where action ordering, object involvement, and interaction geometry vary significantly across executions.

By Franziska Herbert, Vignesh Prasad, Han Liu, Dorothea Koert, Georgia Chalvatzaki

llms robotics fine-tuning multimodal

Related stories