The paper presents a multimodal framework that aligns open-ended learning autonomous robots with human purposes using large language models (LLMs) and vision-language models (VLMs). It transforms human intentions expressed in natural language into robot missions and drives through a three-layer architecture—semantic alignment, mission generation, and drive modeling—allowing bidirectional interaction. Experiments in simulation and on a physical robot show that the system generates valid missions and behaviors, maintaining alignment with human intent.
arXiv:2606. 17073v1 Announce Type: cross Abstract: While commonsense knowledge may suffice for virtual agents, embodied robots interacting with humans require grounded and semantically rich representations of both their environment and their own physical embodiment.
arXiv:2608. 16806v1 Announce Type: cross Abstract: Large Language Models (LLMs) have demonstrated capabilities in in-context learning, task decomposition, step-by-step reasoning, and code generation, driving their gradual evolution from text generation models into the core of agents capable of perceiving environments, invoking tools, and executing tasks.
arXiv:2607. 03461v1 Announce Type: cross Abstract: World models aim to capture environment dynamics in ways that support perception, reasoning, and action, and have recently become a central direction in Vision-Language-Action-World (VLAW) modeling.
The paper introduces a ROS-Agent architecture that enhances task reliability and execution efficiency for open‑source LLM‑powered robotic agents. It adds a MetaTool that forces the LLM to produce a structured pseudo‑code plan before any action, storing this plan in a scratchpad to separate planning from execution. Experiments on a custom mobile robot show up to ~24% improvement in complex task completion and contextual consistency compared to the baseline.
arXiv:2605. 30280v2 Announce Type: replace-cross Abstract: Embodied intelligence is often studied through specialized models for individual tasks such as manipulation or navigation, resulting in fragmented capabilities and limited generalization across tasks, environments, and robot embodiments.
arXiv:2512. 19178v2 Announce Type: replace-cross Abstract: Bridging the gap between natural language commands and autonomous execution in unstructured environments remains an open challenge for robotics.
The paper introduces STEP, a State‑Aware Task Estimator and Planner that uses multi‑modal large language models to explicitly estimate system states and predict state transitions during task planning. By forecasting future states alongside actions, STEP reduces hallucinated actions and improves task‑convergent planning. In a simulated robot assembly task, STEP outperforms the state‑of‑the‑art by 32.8% in action executability and 14.8% in final‑state error.
KnowDemo is a framework that generates diverse robot demonstrations from human videos by leveraging structured manipulation knowledge. It uses a vision‑language model to extract task requirements and permissible execution variations, then resolves these against target‑scene entities to guide candidate generation and screening before motion planning. The resulting demonstrations feature multimodal behavior, alternative contact strategies, and valid subtask orders, and have been shown to improve planning success and enable sim‑to‑real policy transfer across three tasks.
The paper introduces a multimodal in‑context learning framework that uses contrastive demonstration modeling to align large language models’ responses with the required reasoning paths. By contrasting suboptimal and better responses and incorporating a response‑conditioned retrieval mechanism, the method explicitly guides models beyond surface imitation. Experiments on various multimodal tasks, especially visual question answering, show consistent performance gains.
The paper introduces iAm.md, a Markdown-based framework that enables robot skill self-assessment through agentic introspection. By mapping open‑vocabulary semantic detections and object segmentations to persistent records, the system anchors autonomous behavior generation in deployment evidence, mitigating grounding failures. Experiments on a simulated TIAGo robot demonstrate that this representation supports both skill self‑assessment and executable task generalization for navigation‑and‑manipulation tasks.
arXiv:2609.23997v1 Announce Type: cross Abstract: Multi-robot collaboration could enable more efficient and scalable solutions to complex robotic tasks, but collaboration under partial observability...
The paper introduces ProAction, a multimodal dataset of 10,000 samples comprising visual, audio, and text inputs across 12 daily-life scenarios, designed to support the Proactive Robot Action Reasoning (ProRobo) problem. It presents a two-stage human-in-the-loop annotation pipeline that incorporates appraisal and Theory-of-Mind considerations to generate cognitively grounded high-level action labels. The authors benchmark multimodal large language models and propose MMC2Act, showing that training on ProAction significantly improves proactive action reasoning compared to general-purpose models.