arXiv AI
Compositional Unalignment: Auditing Safety in Large Vision-Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 19
Learning to Reason About Implicit Multimodal Harm: From Diagnosis to Generalization
The paper introduces MuPHI, a diagnostic benchmark that isolates multimodal harm arising solely from image‑text composition, providing benign counterfactuals and annotated harm rationales. Evaluation with MuPHI shows that vision‑language models can ground individual modalities but struggle to bind evidence into a coherent harm mechanism, and that existing harm detection methods transfer poorly to new harm distributions. To address this, the authors propose Compositional Harm Interpretation Learning (CHIL), which rewards evidence identification, cross‑modal binding, and verdict consistency, leading to significant improvements in out‑of‑distribution transfer and counterfactual robustness.
arXiv AI
Jun 2
Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning
arXiv:2503. 11832v5 Announce Type: replace Abstract: Recent vision language models (VLMs) have made remarkable strides in generative modeling with multimodal inputs, particularly text and images.
arXiv AI
Aug 24
RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs
arXiv:2605.01913v2 Announce Type: replace-cross Abstract: Fine-tuning safety-aligned language models for downstream tasks often leads to substantial degradation of refusal behavior, making models vul...
arXiv Computation and Language
Sep 4
Towards Multi-modal Multi-turn Safety: From Agentic Interaction to Strategic Alignment
The paper introduces MINT‑Safe, a new open‑source dataset of 11,270 multi‑image dialogues and 500 refusal VQA pairs designed to expose safety risks in multi‑modal large language models during open‑ended conversations. It also proposes TAD‑Align, a turn‑aware dual‑objective reward framework that dynamically up‑weights dialogue turns with inconsistent safety behavior, improving safety metrics on Qwen2.5‑VL‑7B‑Instruct and LLaVA‑Next‑7B. The results show over 10% reduction in attack success rate and notable gains in harmlessness and helpfulness while maintaining overall model performance.
arXiv AI
Aug 6
CARGO-VL: Counterfactual Arbitration with Risk-Constrained Group Optimization for Vision-Language Models
arXiv:2608. 04509v1 Announce Type: new Abstract: Vision-language systems combine images with retrieved text, but these sources can disagree or jointly fail to support an answer.
Hugging Face Trending Papers
Jun 17
The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMs
Warning: This paper studies stereotypes and biases, and contains potentially disturbing examples, used for illustration purposes only. Our findings should not be interpreted as an argument against alignment.