arXiv Computer Vision

DeicticVLA: Unifying Instruction Modes Based on Language and Deictic Gestures in a Single VLA

DeicticVLA unifies three instruction modes—Language Instruction, Vision‑Language Instruction, and Visual Instruction—into a single text prompt and deictic mask framework, allowing a single pretrained Vision‑Language‑Action model to handle all modes. The approach uses text‑prompt completion and deictic gesture grounding, and evaluates various visual prompting methods and training strategies in simulation and real‑world tasks. Results show that two‑stage training improves deictic mask usage, and that Vision‑Language and Visual Instruction outperform Language Instruction on unseen expressions, appearance changes, and novel objects, achieving 100% success on unseen categories.

arXiv Computer Vision

Sep 11

Prompting with Sign Parameters for Low-resource Sign Language Instruction Generation

The paper introduces BdSLIG, the first Bengali Sign Language Instruction Generation dataset, aimed at evaluating Vision Language Models on under-resourced SLIG tasks and long-tail visual concepts. It proposes Sign Parameter-Infused (SPI) prompting, which embeds standard sign parameters such as hand shape, motion, and orientation into textual prompts to improve zero-shot performance and produce more structured, reproducible instructions. The work seeks to promote inclusivity and advance sign language learning systems for under-resourced communities.

By Md Tariquzzaman, Md Farhan Ishmam, Saiyma Sittul Muna, Md Kamrul Hasan, Hasan Mahmud

llms multimodal

arXiv Machine Learning

Jun 11

Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering

arXiv:2606. 12299v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models provide a natural language interface to robot control, but the mapping from language to behavior is often brittle and unintuitive: semantically similar instructions can induce drastically different behaviors, while some capabilities may not be elicitable through prompting alone.

By Hyun Joe Jeong, Gokul Swamy, Andrea Bajcsy

llms robotics fine-tuning multimodal

arXiv AI

Jul 3

Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration

arXiv:2603. 06001v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models enable robots to perform manipulation tasks directly from natural language instructions and are increasingly viewed as a foundation for generalist robotic policies.

By Ninghao Zhang, Bin Zhu, Shijie Zhou, Jingjing Chen

robotics multimodal benchmarks safety

Hugging Face Trending Papers

Jul 29

TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM

Vision-language-action (VLA) models commonly adopt an LLM-centric $V \to L \to A$ pathway, where visual observations are projected into the representation space of a large language model before being decoded into robot actions. Although effective, this design incurs substantial computation and memory overhead at every policy invocation.

llms robotics efficiency multimodal

arXiv AI

Jun 19

Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting

arXiv:2512. 20014v3 Announce Type: replace-cross Abstract: While Vision-Language-Action (VLA) models generalize well to generic instructions, they struggle with personalized commands such as "bring my cup," where the robot must act on one specific instance among visually similar objects.

By Sangoh Lee, Sangwoo Mo, Wook-Shin Han

llms rag robotics multimodal benchmarks

arXiv Machine Learning

Sep 11

Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models

The paper explores soft prompting for few‑shot object detection with vision‑language models, showing that optimizing a small number of continuous prompt tokens—especially when placed at the cross‑modal boundary and initialized from an empty space token—can match LoRA performance while training far fewer parameters. Soft prompting also avoids catastrophic forgetting, transfers to newer models, and can be verbalized into readable prompts. The study extends these findings to manipulation tasks, indicating that VLMs already contain much of the necessary knowledge for specialized domains, and the main challenge is learning how to ask for it.

By Gautam Rajendrakumar Gare, Siyi Li, Hewei Wang, Cesar Daniel Hernandez, Wei Zhao, Wolfgang M. Pauli, John Galeotti, Deva Ramanan

llms computer-vision robotics fine-tuning multimodal

arXiv AI

Jul 7

VLA Grounder: Language-Conditioning Space Optimization for Black-Box VLA Models

arXiv:2607. 04517v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models are commonly treated as end-to-end action policies conditioned on natural-language task descriptions.

By Damir Shodiev, Aleksei Staroverov, Nikita Kachaev, Alexey K. Kovalev, Aleksandr I. Panov

reinforcement-learning robotics multimodal

arXiv AI

Jul 17

Action QFormer: Structured Representation Shaping under Action Supervision in Vision-Language-Action Models

arXiv:2607. 14635v1 Announce Type: new Abstract: Action supervision in vision-language-action (VLA) models is often treated as a downstream objective for learning action prediction.

By Yufeng Ji, Wenhao Tang, Haoyi Niu, Koushil Sreenath, Yi Wu, Zhongyu Li

robotics multimodal

arXiv AI

Jul 14

TS-Mask VLA: 2D Temporal-Spatial Masking for Vision-Language-Action Model with Effective Bridging

arXiv:2607. 09818v1 Announce Type: cross Abstract: Vision-language-action (VLA) models aim to understand natural-language instructions and visual observations, and to generate and execute corresponding actions as embodied agents.

By Shengzhuo Yang, Ronghao Yu, Chuanjie Lv, Linpeng Peng, Hang Yu, Jie Ren, Jiajun Lv, Yong Liu

agents diffusion robotics multimodal benchmarks

arXiv Machine Learning

Jul 9

Pelican-VLA 0.5: Attending Before Acting Benefits Generalization

arXiv:2607. 06655v1 Announce Type: cross Abstract: In this report, we present Pelican-VLA 0.

By Zeyuan Ding, Wenhai Liu, Yang Xu, Jiayu Hu, Yinda Chen, Yi Zhang, Yong Dai, Jian Tang, Xiaozhu Ju

computer-vision robotics fine-tuning multimodal

arXiv AI

Jul 9

T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs

arXiv:2511. 16107v3 Announce Type: replace-cross Abstract: Visual in-context learning (VICL) solves visual tasks by conditioning on a few input-output demonstrations without any model training.

By Shao-Jun Xia, Huixin Zhang, Zhengzhong Tu

llms multimodal safety

arXiv AI

Jun 24

InSight: Self-Guided Skill Acquisition via Steerable VLAs

arXiv:2606. 24884v1 Announce Type: cross Abstract: Vision-language-action (VLA) models can learn manipulation skills from demonstrations, but their capabilities are bounded by the skills in the training data.

By Maggie Wang, Lars Osterberg, Stephen Tian, Ola Shorinwa, Jiajun Wu, Mac Schwager

agents computer-vision robotics multimodal

Related stories