arXiv AI

An Effective Router for Vision-Language Model Selection

arXiv:2606. 08970v1 Announce Type: new Abstract: Vision-language models (VLMs) with varying performance and resource requirements are widely deployed, making it difficult for users to select the most appropriate one among numerous VLM candidates.

arXiv AI

Jun 17

Mordal: Automated Pretrained Model Selection for Vision Language Models

arXiv:2502. 00241v2 Announce Type: replace-cross Abstract: Incorporating multiple modalities into large language models (LLMs) is a powerful way to enhance their understanding of non-textual data, enabling them to perform multimodal tasks.

By Shiqi He, Insu Jang, Mosharaf Chowdhury

llms robotics multimodal benchmarks

arXiv Machine Learning

Jun 30

DataComp-VLM: Improved Open Datasets for Vision-Language Models

arXiv:2606. 28551v1 Announce Type: cross Abstract: Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies.

By Matteo Farina, Vishaal Udandarao, Thao Nguyen, Selim Kuzucu, Maximilian B\"other, Andreas Hochlehnert, Adhiraj Ghosh, Marianna Nezhurina, Karsten Roth, Joschka Struber, Yuhui Zhang, Sebastian Dziadzio, Elaine Sui, Soumya Jahagirdar, Dhruba Ghosh, Hasan Hammoud, Thomas De Min, Simone Caldarella, Jehanzeb Mirza, Sedrick Keh, Mehdi Cherti, Hilde Kuehne, Bernt Schiele, Serena Yeung-Levy, Muhammad Ferjad Naeem, Federico Tombari, Ana Klimovic, Elisa Ricci, Matthias Bethge, Sewoong Oh, Ameya Prabhu, Alessio Tonioni, Jenia Jitsev, Massimiliano Mancini, Ludwig Schmidt, Nikhil Parthasarathy

llms multimodal benchmarks

arXiv AI

5d ago

MLLM-Routed Heterogeneous Ensembles for Robust Cross-Dataset Image Classification

arXiv:2608. 13463v1 Announce Type: cross Abstract: Modern image classification models excel when trained on single task-specific datasets but often struggle to generalize across domains and difficulty levels.

By Daniel Perkins, John Squires, Janou Milligan, Chandra Raskoti, Linda Ungerboeck

llms agents computer-vision robotics multimodal safety

arXiv AI

Jun 2

VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models

arXiv:2601. 03309v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models, which integrate pretrained large Vision-Language Models (VLM) into their policy backbone, are gaining significant attention for their promising generalization capabilities.

By Jianke Zhang, Xiaoyu Chen, Qiuyue Wang, Mingsheng Li, Yanjiang Guo, Yucheng Hu, Jiajun Zhang, Shuai Bai, Junyang Lin, Jianyu Chen

llms computer-vision robotics fine-tuning multimodal benchmarks

arXiv AI

Jul 15

ReLope: KL-Regularized LoRA Probes for Multimodal LLM Routing

arXiv:2603. 24787v2 Announce Type: replace Abstract: Routing has emerged as a promising strategy for balancing performance and cost in large language model (LLM) systems that combine lightweight models with powerful but expensive large models.

By Yaopei Zeng, Congchao Wang, Blake JianHang Chen, Lu Lin

llms fine-tuning multimodal

arXiv AI

Jul 31

Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution

arXiv:2607. 26596v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual understanding within a unified transformer architecture.

By Mingkuan Feng, Zhengqi Wen, Jianhua Tao

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 19

The Hidden Evolution of Disguised Visual Context inside the VLM

arXiv:2606. 20077v1 Announce Type: cross Abstract: Visual tokens enter Large Language Models (LLMs) as raw, foreign signals.

By Wish Suharitdamrong, Tony Alex, Muhammad Awais, Sara Atito

llms multimodal benchmarks

arXiv AI

Jun 11

AVIS: Adaptive Test-Time Scaling for Vision-Language Models

arXiv:2606. 11576v1 Announce Type: cross Abstract: Modern Vision-Language Models (VLMs) benefit from chain-of-thought prompting and test-time scaling, but these gains often come with prohibitive inference cost due to large visual contexts and long decoding chains.

By Ahmadreza Jeddi, Minh Ngoc Le, Amirhossein Kazerouni, Hakki Can Karaimer, Hue Nguyen, Iqbal Mohomed, Michael Brudno, Alex Levinshtein, Konstantinos G. Derpanis, Babak Taati, Radek Grzeszczuk

llms reinforcement-learning efficiency multimodal benchmarks

arXiv Machine Learning

Jul 22

Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning

arXiv:2607. 18615v1 Announce Type: cross Abstract: Machine unlearning for vision-language models (VLMs) remains underexplored.

By Zijie Liu, Jinhao Duan, Gaowen Liu, Sijia Liu, Tianlong Chen

llms multimodal

arXiv Machine Learning

6d ago

LLM Router: Rethinking Routing with Prefill Activations

arXiv:2603. 20895v3 Announce Type: replace-cross Abstract: Existing routers rely on semantic query features or handcrafted features, which often fail to capture model-specific failures or intrinsic task difficulty.

By Tanay Varshney, Annie Surla, Michelle Xu, Gomathy Venkata Krishnan, Maximilian Jeblick, David Austin, Neal Vaidya, Davide Onofrio

llms

arXiv AI

Jun 11

Does the Question Really Matter? Training-Free Data Selection for Vision-Language SFT

arXiv:2603. 09715v2 Announce Type: replace Abstract: Visual instruction tuning is crucial for improving vision-language large models (VLLMs).

By Peng Sun, Yi Yang, Huawen Shen, Yi Ban, Tianfan Fu, Yanbo Wang, Yuqiang Li

fine-tuning multimodal

arXiv Machine Learning

Aug 10

Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models

arXiv:2608. 06901v1 Announce Type: cross Abstract: Vision-language models (VLMs) have achieved remarkable generalization across diverse multimodal tasks through large-scale pre-training, yet their rapidly increasing computational and memory requirements pose significant challenges for deployment in constrained environments.

By Minseok Kang, Hyunwoo Kim, Chanyoung Kim, Minwoo Kim, Jaekoo Lee, Dahuin Jung

llms efficiency multimodal

Related stories