arXiv Machine Learning

Not All Layers Need Tuning: Diagnosing and Directing Adaptation in Vision-Language-Action Models

arXiv Computer Vision
Sep 7

Cross-Domain Tracker Adaptation Without Target-Domain Labels via Vision-Language Agents

The paper introduces a Vision‑Language Model (VLM) that acts as a diagnostic agent to adapt a detect‑to‑track system to new domains without target‑domain labels. By inspecting rendered tracking outputs, the VLM identifies failure modes and iteratively recommends parameter updates, recovering a significant portion of performance lost when transferring hyperparameters from a source domain. Experiments on MOT17→MOT20 show the VLM tuner restores 67.8% of the lost headroom, while Bayesian optimization with proxy objectives performs poorly under large domain shifts.

By Daniel Davila, Ravikumar Balakrishnan, Mike Cochran
arXiv AI
Aug 25

What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs

The paper introduces the Capability-Driven Multimodal Scaling Law, a cross-family framework that predicts vision-language model (VLM) benchmark accuracy from a low-dimensional textual capability score extracted via PCA. By training over 150 VLMs on 34 large language models across seven families, the authors demonstrate that the law accurately extrapolates transfer rates from 8B to 72B‑parameter backbones, predicts full training trajectories, and generalizes to unseen model families. The study also reveals actionable insights, such as certain textual benchmarks negatively correlating with multimodal performance and base LLMs outperforming instruction-tuned counterparts as VLM backbones due to higher absorption rates.

By Ziran Li, Qiang Wang, Zhengyu Chen, Shanglin Lei, Borun Chen, Jingang Wang, Xunliang Cai
arXiv AI
Jun 2

VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models

arXiv:2601. 03309v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models, which integrate pretrained large Vision-Language Models (VLM) into their policy backbone, are gaining significant attention for their promising generalization capabilities.

By Jianke Zhang, Xiaoyu Chen, Qiuyue Wang, Mingsheng Li, Yanjiang Guo, Yucheng Hu, Jiajun Zhang, Shuai Bai, Junyang Lin, Jianyu Chen
arXiv Computer Vision
Sep 15

What Makes an Efficient VLA? Navigating Action-Head Design, Scaling, and Latency

arXiv:2609.13984v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models combine a pretrained vision encoder, a language backbone, and an action head, but their relative contribution has...

By Luoyang Sun, Guoyang Xia, Fengfa Li, Lei Ren, Xinyu Cui, Haifeng Zhang, Fangxiang Feng, Kaike Zhang, Kun Zhan, Yan Xie, Jun Wang, Cheng Deng
arXiv Computer Vision
Sep 24

Beyond Balanced Accuracy: A Resolution and Parity-Controlled Benchmark for Vision-Language and Vision-Only Defect Assessment in UAV Power-Line Inspection

The paper evaluates the claim that vision‑language models (VLMs) outperform task‑specific vision backbones for UAV power‑line defect assessment using the ElecVQA‑Bench benchmark. Across various evaluation settings—partitioning, item sets, label spaces, replication, resolution, and side information—the performance gap between VLMs and traditional backbones is minimal or even reversed when controlling for resolution and token budget. The study concludes that VLM superiority is not universally supported and emphasizes the importance of rigorous benchmark audits.

By Linghao Zhang, Siyu Xiang, Junwei Kuang, Peiyu Yi