arXiv Machine Learning September 11, 2026 By Gautam Rajendrakumar Gare, Siyi Li, Hewei Wang, Cesar Daniel Hernandez, Wei Zhao, Wolfgang M. Pauli, John Galeotti, Deva Ramanan

Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models

Read the original on arXiv Machine Learning →

The paper explores soft prompting for few‑shot object detection with vision‑language models, showing that optimizing a small number of continuous prompt tokens—especially when placed at the cross‑modal boundary and initialized from an empty space token—can match LoRA performance while training far fewer parameters. Soft prompting also avoids catastrophic forgetting, transfers to newer models, and can be verbalized into readable prompts. The study extends these findings to manipulation tasks, indicating that VLMs already contain much of the necessary knowledge for specialized domains, and the main challenge is learning how to ask for it.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 11

Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering

arXiv:2606. 12299v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models provide a natural language interface to robot control, but the mapping from language to behavior is often brittle and unintuitive: semantically similar instructions can induce drastically different behaviors, while some capabilities may not be elicitable through prompting alone.

By Hyun Joe Jeong, Gokul Swamy, Andrea Bajcsy

llms robotics fine-tuning multimodal

arXiv AI

Jun 2

VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models

arXiv:2601. 03309v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models, which integrate pretrained large Vision-Language Models (VLM) into their policy backbone, are gaining significant attention for their promising generalization capabilities.

By Jianke Zhang, Xiaoyu Chen, Qiuyue Wang, Mingsheng Li, Yanjiang Guo, Yucheng Hu, Jiajun Zhang, Shuai Bai, Junyang Lin, Jianyu Chen

llms computer-vision robotics fine-tuning multimodal benchmarks

arXiv Computer Vision

Oct 23

Watch and Learn: Teaching Visual In-Context Learning by Removing Textual Shortcuts

The paper introduces MIME, a task‑agnostic visual in‑context learning (vICL) dataset that removes textual shortcuts by omitting explicit rules linking demonstration images to answers, forcing models to infer the rule visually. Fine‑tuning vision‑language models on MIME boosts vICL accuracy by 8.1–18.0 points across three VLMs while preserving single‑image performance and improving multi‑image and text‑only in‑context learning. The authors also propose DTAD, a technique that randomly drops demonstration labels during training to encourage visual inference, achieving comparable accuracy with more consistent attention to demonstration images and outperforming frontier models on the TrueMICL benchmark.

By Nimrod Shabtay, Asaf Yehudai, Avihu Dekel, Eli Schwartz, Raja Giryes

llms fine-tuning multimodal

arXiv AI

Oct 16

EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training

EmbodiedMidtrain bridges the gap between Vision‑Language Models (VLMs) and Vision‑Language‑Action Models (VLAs) by first analyzing the distribution mismatch between their data. It introduces a lightweight proximity estimator that selects VLA‑aligned samples from a large VLM pool, creating the EmbodiedMidtrain‑1M dataset used to mid‑train VLMs before VLA fine‑tuning. Experiments on simulated and real‑world robot tasks show consistent performance gains across different VLM backbones and VLA action heads, with early training steps already benefiting from the improved initialization.

By Yiyang Du, Zhanqiu Guo, Haochen Zhang, Annabella Macaluso, Xin Ye, Liu Ren, Yonatan Bisk, Chenyan Xiong

llms robotics fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Aug 3

Visual Distribution Anchoring for Efficient Prompt Tuning

arXiv:2607. 28967v1 Announce Type: cross Abstract: Prompt tuning adapts vision--language models with few trainable parameters, but existing approaches trade off efficiency and adaptation: static textual prompts can overfit source classes, image-conditioned prompts add per-instance computation, and multimodal tuning modifies the visual branch.

By Pouya Parsa, Raoof Zare Moayedi, Seongjin Choi

llms multimodal

arXiv Computer Vision

5d ago

Vision-Language-Action or World-Action? Just Action Models

The paper introduces the Just Action Model (JAM), a framework that decouples backbone initialization, future observation objectives, and inference machinery to unify vision-language-action (VLA) and world-action model (WAM) approaches. JAM employs a pretrained visual backbone, an action expert, an action loss, and a foresight loss that predicts future encodings without visual generation, discarding the prediction head after training. Experiments show JAM outperforms FastWAM and ImageWAM on LIBERO, LIBERO-Plus, and RoboTwin, while reducing online training FLOPs by 73%.

By Yonghao Yu, Lang Huang, Runyi Li, Toshihiko Yamasaki