arXiv AI September 7, 2026 By Minji Kim, Jihyoung Jang, Hyounghun Kim

Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

The paper introduces KoNA, a benchmark designed to evaluate selective non‑compliance in vision‑language models (VLMs) across five categories—False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety. KoNA tests both query‑level and component‑level non‑compliance using paired single and compound queries, revealing that many VLMs struggle to refuse, correct, or abstain appropriately, especially when selective non‑compliance is required. Fine‑tuning VLMs on KoNA examples improves non‑compliance accuracy while preserving performance on fully answerable tasks, indicating that models can better distinguish answerable components from those needing non‑compliance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Sep 7

Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models

The paper investigates how large language models balance helpfulness and safety by refusing harmful queries while responding to benign ones. It decomposes safety-tuning responses into a boilerplate refusal statement and a rationale, finding that the statement causes false refusals by relying on superficial cues. Training on rationales alone reduces false refusals without compromising safety performance, suggesting that fine‑grained safety supervision is essential for better alignment.

By Minji Kim, Hyounghun Kim

llms agents safety

arXiv Computation and Language

Aug 21

Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System Messages

arXiv:2608. 19207v1 Announce Type: new Abstract: Production deployments of Multimodal Large Language Models (MLLMs) increasingly rely on system messages to govern model behavior.

By Juan Yeo, Geewook Kim

llms multimodal benchmarks

arXiv AI

4d ago

Suppression Is Not Forgetting: Residual Recoverability in Visual Concept Unlearning for VLMs

arXiv:2604.03114v2 Announce Type: replace-cross Abstract: Vision-language models (VLMs) may need to forget visual concepts after deployment because of privacy, copyright, licensing, safety, or policy...

By Zhangyun Tan, Zeliang Zhang, Jiani Liu, Susan Liang, Yolo Y. Tang, Lisha Chen, Chenliang Xu

llms multimodal safety

arXiv AI

Jul 8

Modality Relevance is not Modality Utility: Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAG

arXiv:2607. 05438v1 Announce Type: cross Abstract: Multimodal retrieval-augmented generation (RAG) grounds a generator in evidence drawn from heterogeneous modalities -- text, tables, and images.

By Xue Li, Yiming Gai

llms rag multimodal

arXiv AI

Jun 26

Know2Guess: A Contamination-Aware Multi-Zone Benchmark for Knowledge-Boundary Evaluation in Large Language Models

arXiv:2606. 26101v1 Announce Type: cross Abstract: Reliable evaluation of large language models should separate supported answering from unsupported guessing without conflating either with data contamination, prompt idiosyncrasy, or generic refusal behavior.

By Renwei Meng, Bowen Zhang, Jian Wang, Xican Wang, Haoyi Wu, Xuanyan Qiu, Shengan Yang

llms benchmarks

arXiv AI

Jul 21

RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants

arXiv:2607. 16243v1 Announce Type: cross Abstract: Multimodal industrial anomaly inspection assistants are a critical component of next-generation smart factories, enabling interactive vision-language-based querying.

By Anushiya Arunan, Xin Li, Yan Qin, U-Xuan Tan, Nhu Khue Vuong, Xiaoli Li, Chau Yuen

llms multimodal benchmarks safety

One email a morning, machine-written

Related stories

Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models

Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System Messages

Suppression Is Not Forgetting: Residual Recoverability in Visual Concept Unlearning for VLMs

Modality Relevance is not Modality Utility: Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAG

Know2Guess: A Contamination-Aware Multi-Zone Benchmark for Knowledge-Boundary Evaluation in Large Language Models

RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants