The paper investigates whether vision‑language‑action (VLA) models truly follow How‑level instructions that detail how to perform a task. The authors find that current VLAs do not benefit from such instructions and that task success actually decreases. They propose training with segment‑level enriched instructions and a new benchmark, SIMPLER‑IN, to better assess How‑level instruction following, but conclude that this remains an open challenge.
By Sangwu Park, Wonjoong Kim, Yeonjun In, Chanyoung Park
arXiv:2606. 12299v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models provide a natural language interface to robot control, but the mapping from language to behavior is often brittle and unintuitive: semantically similar instructions can induce drastically different behaviors, while some capabilities may not be elicitable through prompting alone.
By Hyun Joe Jeong, Gokul Swamy, Andrea Bajcsy
arXiv:2603. 06001v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models enable robots to perform manipulation tasks directly from natural language instructions and are increasingly viewed as a foundation for generalist robotic policies.
By Ninghao Zhang, Bin Zhu, Shijie Zhou, Jingjing Chen
arXiv:2610.10526v1 Announce Type: cross
Abstract: Vision-language-action models (VLAs) are strikingly sensitive to instruction phrasing and do not inherit the language robustness of the vision-langua...
By Mikey Watts (Independent Researcher), Yuchen Cui (University of California, Los Angeles)
DeicticVLA unifies three instruction modes—Language Instruction, Vision‑Language Instruction, and Visual Instruction—into a single text prompt and deictic mask framework, allowing a single pretrained Vision‑Language‑Action model to handle all modes. The approach uses text‑prompt completion and deictic gesture grounding, and evaluates various visual prompting methods and training strategies in simulation and real‑world tasks. Results show that two‑stage training improves deictic mask usage, and that Vision‑Language and Visual Instruction outperform Language Instruction on unseen expressions, appearance changes, and novel objects, achieving 100% success on unseen categories.
By Kango Yanagida, Tatsuya Aoki, Yuichiro Yoshikawa, Takato Horii
The paper introduces CRAFT, a method for improving compositional generalization in vision‑language‑action (VLA) models. It addresses the issue where models rely on visual shortcuts during fine‑tuning, leading them to execute demonstrated skill combinations that match observations rather than the instructed ones. By training with counterfactual instruction–observation pairs and transferring supervision through reusable skill representations, CRAFT enhances success on unseen skill combinations while preserving performance on demonstrated ones across multiple VLA models and benchmarks.
By Taegeun Yang, Youngju Na, Yoonki Cho, Sung-Eui Yoon
The paper investigates a specific failure mode in vision‑language‑action models, termed instruction‑action binding, where models respond to language and vision separately but fail to combine them to select the correct action under counterfactual changes. Through behavioral analyses of fine‑tuned policies, the authors show that failed rollouts often preserve source behavior or switch to other demonstrated tasks, indicating that language is not ignored but mis‑bound. They propose Equivariant Counterfactual Training (ECT), which supplies counterfactual demonstrations and a paired loss to enforce correct action selection, achieving significant performance gains across simulated and real‑world benchmarks.
By Hung-Jen Chen, Yu-Hsun Hou, Yan-Hong Chen, Yan-Fu Chen, Binghua Cai, Min Sun, Chun-Yi Lee
arXiv:2607. 02222v1 Announce Type: cross Abstract: Vision-Language Navigation has increasingly emphasized high-level instruction reasoning, memory, global map construction, and instruction decomposition, while the low-level action representation remains comparatively underexplored.
By Haokun Liu, Zhaoqi Ma, Yicheng Chen, Wentao Zhang, Masaki Kitagawa, Zicen Xiong, Jinjie Li, Moju Zhao
arXiv:2606. 09630v1 Announce Type: cross Abstract: Vision-language-action (VLA) policies provide strong priors for language-conditioned manipulation, but remain brittle in off-nominal states requiring targeted recovery.
By Haodi Hu, Chung-Ta Huang, Jing Liu, Ye Wang, Kei Suzuki, Matthew Brand, Toshiaki Koike-Akino
arXiv:2610.34028v1 Announce Type: new
Abstract: As Vision-Language Models (VLMs) increasingly enable robots to follow language instructions, understanding their navigation decisions becomes important...
By Yiyuan Yao, Xinpeng Ding, Minglei Chen, Fei Gao, Lihuo He, Jie Yuan, Aijun Li, Nannan Wang, Xinbo Gao
arXiv:2606. 27755v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models enable instruction-driven robotic manipulation, but they inherit oversized language backbones from pretrained VLMs whose capacity far exceeds what is needed for short robotic instructions.
By Guoheng Sun, Kaixi Feng, Shwai He, Xiaochuan Gong, Yexiao He, Ziyao Wang, Zheyu Shen, Wanghao Ye, Ramana Rao Kompella, Gaowen Liu, Ang Li
arXiv:2606.04588v2 Announce Type: replace
Abstract: Multimodal large language models have made rapid progress in video understanding, yet existing benchmarks largely rely on simple prompts and provid...
By Huangchen Xu, Yuan Wu, Yi Chang