DiaVLo is a diagnostic framework for vision‑language models (VLMs) that uses human curation and VLM generation to create specifications of desired and observed behaviours, revealing potential misalignments. It also offers causal estimates to pinpoint the most influential concepts driving VLM behaviour. Experiments on several open‑source VLMs under classification and generation tasks show that DiaVLo’s behaviour labels correlate with model performance and illuminate how VLMs perceive, organise, and prioritise concepts.
arXiv:2602. 18094v2 Announce Type: replace-cross Abstract: Existing Visual-Language Models (VLMs) have achieved significant progress by being trained on massive-scale datasets, typically under the assumption that data are independent and identically distributed (IID).
arXiv:2609.24576v1 Announce Type: cross Abstract: Modern Vision-Language Navigation (VLN) models rely mostly on pre-trained large Vision-Language Models (VLMs) to predict navigation actions. While th...
arXiv:2602.06652v2 Announce Type: replace Abstract: The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions r...
arXiv:2607. 09544v1 Announce Type: cross Abstract: Despite strong performance on many multimodal tasks, vision-language models (VLMs) still struggle with basic object counting.
arXiv:2608. 07435v1 Announce Type: cross Abstract: Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind, making weaknesses hard to identify.
The paper investigates using Vision Language Models (VLMs) to accelerate verification and validation (V&V) of classification models by automatically detecting systematic errors. It introduces a VLM-based error slice detection (ESD) method that groups and labels errors, demonstrating its ability to identify perturbations in a non-military dataset and to cluster images by surroundings in a military context. The study highlights challenges such as underrepresentation of defence data in VLM training and limited contextual diversity, and suggests that while fully automated V&V is not yet feasible, VLMs could speed up the process in the future.
arXiv:2503. 12999v4 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) have demonstrated exceptional performance in various multi-modal tasks.
arXiv:2606. 18839v1 Announce Type: new Abstract: Vision-language models (VLMs) are now widely used in downstream tasks.
arXiv:2606. 20077v1 Announce Type: cross Abstract: Visual tokens enter Large Language Models (LLMs) as raw, foreign signals.
arXiv:2608.22857v1 Announce Type: new Abstract: Vision-language models (VLMs) frequently fail at visual change reasoning, even when their vision encoders contain sufficient information. We observe th...
arXiv:2608. 12333v1 Announce Type: cross Abstract: Vision-language models must associate visual entities with textual attributes.
FPCO-Dialog is a new benchmark designed to evaluate how vision‑language models correct and cooperate when faced with repeated false premises in multi‑turn dialogues. The dataset contains 1,080 images and 10,800 question turns, organized by visual complexity, object category, and false‑premise class, and follows a 10‑turn protocol where a correct dialogue prefix is followed by repeated false‑premise expressions. Using a model‑agnostic protocol and the CorrTP@K correction‑rate metric, the benchmark reveals significant differences among 20 commercial and open‑source VLMs in their correction tendencies, turn‑wise dynamics, and responses to different false‑premise types.