arXiv AI

Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

The paper introduces KoNA, a benchmark designed to evaluate selective non‑compliance in vision‑language models (VLMs) across five categories—False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety. KoNA tests both query‑level and component‑level non‑compliance using paired single and compound queries, revealing that many VLMs struggle to refuse, correct, or abstain appropriately, especially when selective non‑compliance is required. Fine‑tuning VLMs on KoNA examples improves non‑compliance accuracy while preserving performance on fully answerable tasks, indicating that models can better distinguish answerable components from those needing non‑compliance.

arXiv AI

Sep 7

Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models

The paper investigates how large language models balance helpfulness and safety by refusing harmful queries while responding to benign ones. It decomposes safety-tuning responses into a boilerplate refusal statement and a rationale, finding that the statement causes false refusals by relying on superficial cues. Training on rationales alone reduces false refusals without compromising safety performance, suggesting that fine‑grained safety supervision is essential for better alignment.

By Minji Kim, Hyounghun Kim

llms agents safety

arXiv Computation and Language

Aug 21

Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System Messages

arXiv:2608. 19207v1 Announce Type: new Abstract: Production deployments of Multimodal Large Language Models (MLLMs) increasingly rely on system messages to govern model behavior.

By Juan Yeo, Geewook Kim

llms multimodal benchmarks

arXiv AI

4d ago

Suppression Is Not Forgetting: Residual Recoverability in Visual Concept Unlearning for VLMs

arXiv:2604.03114v2 Announce Type: replace-cross Abstract: Vision-language models (VLMs) may need to forget visual concepts after deployment because of privacy, copyright, licensing, safety, or policy...

By Zhangyun Tan, Zeliang Zhang, Jiani Liu, Susan Liang, Yolo Y. Tang, Lisha Chen, Chenliang Xu

llms multimodal safety

arXiv AI

Jul 8

Modality Relevance is not Modality Utility: Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAG

arXiv:2607. 05438v1 Announce Type: cross Abstract: Multimodal retrieval-augmented generation (RAG) grounds a generator in evidence drawn from heterogeneous modalities -- text, tables, and images.

By Xue Li, Yiming Gai

llms rag multimodal

arXiv AI

Jun 26

Know2Guess: A Contamination-Aware Multi-Zone Benchmark for Knowledge-Boundary Evaluation in Large Language Models

arXiv:2606. 26101v1 Announce Type: cross Abstract: Reliable evaluation of large language models should separate supported answering from unsupported guessing without conflating either with data contamination, prompt idiosyncrasy, or generic refusal behavior.

By Renwei Meng, Bowen Zhang, Jian Wang, Xican Wang, Haoyi Wu, Xuanyan Qiu, Shengan Yang

llms benchmarks

arXiv AI

Jul 21

RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants

arXiv:2607. 16243v1 Announce Type: cross Abstract: Multimodal industrial anomaly inspection assistants are a critical component of next-generation smart factories, enabling interactive vision-language-based querying.

By Anushiya Arunan, Xin Li, Yan Qin, U-Xuan Tan, Nhu Khue Vuong, Xiaoli Li, Chau Yuen

llms multimodal benchmarks safety

arXiv AI

Aug 17

Never the Number: Structural Abstention for AI Systems Whose Answers Are Consumed as Fact

arXiv:2608. 13926v1 Announce Type: new Abstract: Large language models have made natural language interfaces to databases (NLIDB) newly credible, but LLM text-to-SQL systems fail in a way that matters for deployment: a hallucinated column or a mis-aggregated total yields a fluent wrong answer, indistinguishable at the point of use from a right one.

By Zhelun (Allen), Wu

llms agents fine-tuning benchmarks data-engineering

arXiv Machine Learning

Aug 21

Answer-Level Trust Selection for Physical Vision-Language Reasoning

arXiv:2608. 19807v1 Announce Type: new Abstract: Vision-language models (VLMs) can estimate physical quantities such as duration, speed, and acceleration from visual observations, but existing benchmarks primarily assess overall model performance against annotated ground truth.

By Rongyu Yu, Ke Niu, Fengxiang He

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 18

Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models

arXiv:2606. 19297v1 Announce Type: new Abstract: Embodied Vision-Language-Action (VLA) models are typically obtained by fine-tuning powerful pretrained VLMs on robotics data, yet it is unclear how much commonsense and factual knowledge they retain after adaptation.

By Nikita Kachaev, Andrey Moskalenko, Matvey Skripkin, Nikita Kurlaev, Daria Pugacheva, Albina Burlova, Mikhail Kolosov, Denis Shepelev, Andrey Kuznetsov, Elena Tutubalina, Aleksandr I. Panov, Alexey K. Kovalev, Vlad Shakhuro

agents robotics fine-tuning multimodal benchmarks

Hugging Face Trending Papers

Aug 25

DoublesEval: Diagnosing Multi-Agent Tactical Reasoning in Vision-Language Models via Professional Doubles Badminton

Visual Language Models (VLMs) excel at describing visible scene content but struggle to reason about dynamic multi-agent interactions, where action semantics depend on coordinated roles and spatial-te...

llms agents multimodal

arXiv AI

Sep 25

Reasoning Instructions Can Break Answer Decoding in Vision--Language Models

The paper shows that chain‑of‑thought (CoT) instructions can distort evaluation of vision‑language models (VLMs) when a scorer reads answer‑label logits before the model generates a rationale. On ScienceQA, Qwen2.5‑VL‑7B’s accuracy falls from 80.76% to 45.48% under this CoT‑prefix scoring, and most predictions incorrectly pick the first option. Linear probes and free generation recover most of the lost accuracy, indicating that the answer information remains in the hidden states but is missed by the early readout. The authors explain the mismatch with vocabulary and layer diagnostics, noting that probability mass shifts toward continuation tokens while answer information stays linearly accessible in later layers. The effect varies across datasets and models, but the study demonstrates that CoT‑prefix scoring can misrepresent model knowledge unless the requested and scored outputs are aligned.

By Zeyan Li, Siyuan Qiu, Jianfeng Xu

llms multimodal

arXiv Computer Vision

Sep 2

Beyond Blind Compliance: Benchmarking Task Verification in OCR Reasoning

arXiv:2609.00232v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have achieved strong performance on OCR-centric document understanding and text-rich visual reasoning benchmar...

By Yue Zhou, Yuan Wu, Yi Chang

llms computer-vision multimodal benchmarks

Related stories