arXiv Computer Vision November 6, 2026 By Yu Fang, Yuchun Feng, Dong Jing, Jiaqi Liu, Yue Yang, Zhenyu Wei, Daniel Szafir, Mingyu Ding

When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs

Read the original on arXiv Computer Vision →

Vision‑Language‑Action models (VLAs) often fail to follow language instructions because they rely on vision shortcuts learned from biased datasets. The authors introduce LIBERO‑CF, a counterfactual benchmark that tests language following by providing alternative task instructions for the same scene layouts, revealing widespread counterfactual failures. They propose Counterfactual Action Guidance (CAG), a dual‑branch inference scheme that amplifies language conditioning, which consistently improves language grounding and task success across diverse VLAs without requiring extra data or architectural changes.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

arXiv Machine Learning

Jun 11

Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering

arXiv:2606. 12299v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models provide a natural language interface to robot control, but the mapping from language to behavior is often brittle and unintuitive: semantically similar instructions can induce drastically different behaviors, while some capabilities may not be elicitable through prompting alone.

By Hyun Joe Jeong, Gokul Swamy, Andrea Bajcsy

llms robotics fine-tuning multimodal

arXiv AI

Jul 3

Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration

arXiv:2603. 06001v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models enable robots to perform manipulation tasks directly from natural language instructions and are increasingly viewed as a foundation for generalist robotic policies.

By Ninghao Zhang, Bin Zhu, Shijie Zhou, Jingjing Chen

robotics multimodal benchmarks safety

arXiv Machine Learning

Oct 3

Same Scene, Different Task: Skill Alignment for Compositional Generalization in VLAs

The paper introduces CRAFT, a method for improving compositional generalization in vision‑language‑action (VLA) models. It addresses the issue where models rely on visual shortcuts during fine‑tuning, leading them to execute demonstrated skill combinations that match observations rather than the instructed ones. By training with counterfactual instruction–observation pairs and transferring supervision through reusable skill representations, CRAFT enhances success on unseen skill combinations while preserving performance on demonstrated ones across multiple VLA models and benchmarks.

By Taegeun Yang, Youngju Na, Yoonki Cho, Sung-Eui Yoon

robotics fine-tuning multimodal benchmarks safety

arXiv AI

Jun 16

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

arXiv:2605. 27284v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models are increasingly expected to not only complete robot tasks, but also follow human instructions about how those tasks should be executed.

By Xintong Hu, Xuhong Huang, Jinyu Zhang, Yutong Yao, Yuchong Sun, Qiuyue Wang, Mingsheng Li, Sicheng Xie, Yitao Liu, Junhao Chen, Yixuan Chen, Yingming Zheng, Shuai Bai, Tao Yu

robotics multimodal benchmarks safety

arXiv AI

Aug 19

LoopVLA: Learning Sufficiency in Recurrent Refinement for Vision-Language-Action Models

LoopVLA introduces a recurrent Vision‑Language‑Action architecture that learns to refine multimodal representations, predict actions, and estimate when further refinement is unnecessary. By iteratively applying a shared Transformer block and producing a sufficiency score at each step, it decouples refinement from fixed layer indices and aligns confidence scores with action quality through a self‑supervised objective. Experiments on LIBERO, LIBERO‑Plus, and VLA‑Arena demonstrate that LoopVLA reduces model parameters by 45% and boosts inference throughput up to 1.7× while matching or surpassing strong baselines in task success.

By Boyang Shen, Kaixiang Yang, Hao Wang, Qiuyu Yu, Qiang Xie, Qiang Li, Zhiwei Wang

llms robotics efficiency multimodal safety

arXiv Computer Vision

Oct 1

Correcting WHERE, Preserving HOW: Compositional Generalization for Vision-Language-Action Models via Referential Guidance

arXiv:2609.38616v1 Announce Type: cross Abstract: While Vision-Language-Action (VLA) models enable flexible action generation, their generalization across diverse environmental elements, including ma...

By Yanyan Zhang, Disheng Liu, Xinpeng Li, Chaoda Song, Mohsen Hariri, Debargha Ganguly, Wang Yang, Kai Ye, Bryce Grant, Vipin Chaudhary, Yu Yin

robotics multimodal

One email a morning, machine-written

Related stories

Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering

Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration

Same Scene, Different Task: Skill Alignment for Compositional Generalization in VLAs

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

LoopVLA: Learning Sufficiency in Recurrent Refinement for Vision-Language-Action Models

Correcting WHERE, Preserving HOW: Compositional Generalization for Vision-Language-Action Models via Referential Guidance