The paper introduces BdSLIG, the first Bengali Sign Language Instruction Generation dataset, aimed at evaluating Vision Language Models on under-resourced SLIG tasks and long-tail visual concepts. It proposes Sign Parameter-Infused (SPI) prompting, which embeds standard sign parameters such as hand shape, motion, and orientation into textual prompts to improve zero-shot performance and produce more structured, reproducible instructions. The work seeks to promote inclusivity and advance sign language learning systems for under-resourced communities.
By Md Tariquzzaman, Md Farhan Ishmam, Saiyma Sittul Muna, Md Kamrul Hasan, Hasan Mahmud
Vision-language-action (VLA) models are expected to follow instructions like their language backbones, yet prior work has focused on What-level instructions that specify the task rather than How-level...
arXiv:2606. 12299v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models provide a natural language interface to robot control, but the mapping from language to behavior is often brittle and unintuitive: semantically similar instructions can induce drastically different behaviors, while some capabilities may not be elicitable through prompting alone.
By Hyun Joe Jeong, Gokul Swamy, Andrea Bajcsy
The paper investigates whether vision‑language‑action (VLA) models truly follow How‑level instructions that detail how to perform a task. The authors find that current VLAs do not benefit from such instructions and that task success actually decreases. They propose training with segment‑level enriched instructions and a new benchmark, SIMPLER‑IN, to better assess How‑level instruction following, but conclude that this remains an open challenge.
By Sangwu Park, Wonjoong Kim, Yeonjun In, Chanyoung Park
The paper introduces V‑GIFT, a lightweight method that enhances visual instruction tuning for multimodal large language models by incorporating a small set of visually grounded self‑supervised tasks expressed as natural language instructions. These tasks—such as rotation prediction, color matching, and cross‑view correspondence—are reformulated into image‑instruction‑response triplets, providing supervision that cannot be solved without visual evidence. The approach requires no human annotations, architectural changes, or extra training stages, and consistently improves vision‑centric performance across various models, regimes, and benchmarks when only 3‑10% of the training data is augmented with these instructions.
By Sophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc, Nicolas Thome, Spyros Gidaris
arXiv:2603. 06001v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models enable robots to perform manipulation tasks directly from natural language instructions and are increasingly viewed as a foundation for generalist robotic policies.
By Ninghao Zhang, Bin Zhu, Shijie Zhou, Jingjing Chen