arXiv AI
Jun 16
Tool-IQA: Augmenting Image Quality Assessment with Simple Tools
arXiv:2606. 16082v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have been increasingly adopted for Image Quality Assessment (IQA).
SABRE: Scalable and Automated Benchmarking of VLMs under Stress
arXiv:2608. 07435v1 Announce Type: cross Abstract: Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind, making weaknesses hard to identify.
Related stories
arXiv AI
Aug 14
How Do VLMs Behave When Blind or Misled? Behavioral Evaluation of VLMs on Scientific Figures
arXiv:2608. 13267v1 Announce Type: cross Abstract: Existing vision-language model (VLM) benchmarks emphasize perception and reasoning accuracy (how well VLMs describe and reason about what they see in an image), with limited attention to behavioral reliability under uncertainty (how they behave when visual evidence is missing or misleading).
arXiv AI
Jul 21
Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs
arXiv:2607. 16311v1 Announce Type: cross Abstract: Vision-language models (VLMs) often answer visual questions using learned language and category priors rather than grounding their predictions in the image itself.
arXiv AI
Jul 28
Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization
arXiv:2607. 24354v1 Announce Type: new Abstract: Automatic prompt optimization (APO) has been widely adopted to adapt vision-language models (VLMs) to downstream tasks without weight updates, yielding promising results.
arXiv Computer Vision
Sep 18
Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models
The paper introduces REVEAL, a diagnostic benchmark that stresses Video‑Language Models (VidLMs) on five controlled probes—camera‑motion sensitivity, cross‑frame integration, video sycophancy, language‑only shortcuts, and temporal expectation bias—to assess how well these models encode and use visual evidence. Experiments on 12 VidLMs reveal systematic failures: some visual signals are never reliably encoded, while others are overridden by model priors, leading to performance below chance on several probes that humans solve with high accuracy. Mechanistic probes further pinpoint where and why visual evidence is lost, demonstrating that under assertive prompts a model’s output becomes nearly invariant to real versus random video input, rendering visual evidence causally inert.
arXiv AI
Sep 4
Making Every Tool Call Count: Necessary Tool-Evidence Path Rewards for Agentic Vision-Language Models
The paper introduces the Necessary Tool‑Evidence Path (NTEP) annotation scheme and its associated reward mechanism (NTEP‑R) to better supervise vision‑language models that use external tools. By explicitly specifying which evidence is needed and penalizing redundant tool calls, the authors train an 8B‑parameter model that shows improved accuracy and tool‑use efficiency across seven image‑grounded benchmarks. The approach demonstrates that fine‑grained supervision of tool‑evidence paths is essential for robust agentic VLM performance.
arXiv AI
Sep 10
VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification
arXiv:2609.06245v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) perform strongly on general visual understanding tasks such as visual question answering, yet they often str...
arXiv AI
Jul 14
3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects
arXiv:2607. 10826v1 Announce Type: cross Abstract: Automated evaluation is essential for scaling generative 3D systems, where exhaustive human review is costly and slow.
arXiv AI
Sep 17
EDCT-Bench: Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing
EDCT-Bench is a benchmark that evaluates the faithfulness of Vision‑Language Models (VLMs) by using Explanation‑Driven Counterfactual Testing (EDCT). EDCT extracts visual concepts from a model’s natural language explanation, applies minimal verified edits to those concepts, and checks whether the model’s answer and explanation remain consistent with the edited image. The benchmark covers three domains—knowledge‑intensive VQA, safety‑critical driving, and 3D spatial reasoning—and reveals significant faithfulness gaps in current VLMs, while also showing that EDCT‑generated counterfactuals can improve training.
Hugging Face Trending Papers
Jul 12
3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects
Automated evaluation is essential for scaling generative 3D systems, where exhaustive human review is costly and slow. However, the reliability of an automated judge depends on the entire evaluation pipeline, not only the underlying vision-language model (VLM), but also how assets are rendered, what visual evidence is provided, how the task is specified, and how human reference labels are constructed.
arXiv AI
3d ago
VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenes
arXiv:2605.26380v2 Announce Type: replace-cross Abstract: Frontier multimodal large language models (MLLMs) have been reported to achieve over 90\% accuracy on fine-grained perception benchmarks. How...
arXiv Computer Vision
3d ago
Composition, Not Conversation: VLMs Lose the Scene, Not the Thread
arXiv:2609.38368v1 Announce Type: new Abstract: Vision-language models (VLMs) increasingly reason over visual evidence that is cropped, segmented, retrieved, or revealed over time. Yet most VQA bench...