Hugging Face Trending Papers August 11, 2026

SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learning

Read the original on Hugging Face Trending Papers →

Large vision-language models (LVLMs) remain vulnerable to jailbreak attacks that exploit visual inputs to bypass safety alignment inherited from their language backbones. We propose SafeCap, a reinforcement-learning framework that aligns LVLMs through learned self-captioning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

arXiv AI

Aug 12

SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learning

arXiv:2608. 10513v1 Announce Type: cross Abstract: Large vision-language models (LVLMs) remain vulnerable to jailbreak attacks that exploit visual inputs to bypass safety alignment inherited from their language backbones.

By Caoyuan Ma, Wenpu Liu, Weichu Xie, Tian Gu, Shilei Zhao, Lingxi Min, Shuai Dong, Yuqi Xu, Ji Zhao, Ziyue Wang, Wenzheng Chang, Taiqiang Wu, Yongfu Zhu, Wenqi Shao, Yinqiang Zheng

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Sep 22

Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models

arXiv:2609.22234v1 Announce Type: cross Abstract: Instruction hierarchy (IH) alignment teaches language models to prioritize higher-level instructions when inputs conflict. While studied primarily in...

By Nicholas Sansoterra, Zishuo Zheng, Sachin Kumar

llms agents reinforcement-learning multimodal safety

arXiv AI

Jul 1

Harnessing Textual Refusal Directions for Multimodal Safety

arXiv:2606. 31876v1 Announce Type: new Abstract: To improve safety in Large Language Models (LLMs) we can either perform post-training alignment or exploit refusal directions in the activation space.

By Moreno D'Inc\`a, Massimiliano Mancini, Nicu Sebe

llms multimodal benchmarks safety

arXiv AI

Sep 3

Transfer Safety Awareness for Cross-Modal Safety Drift in Multimodal Large Language Models

The paper investigates cross‑modal safety drift in multimodal large language models, where a harmless text query paired with a visual image can trigger harmful responses. Empirical analysis identifies unsafe response patterns and shows that visual cues receive limited attention, weakening refusal mechanisms. The authors introduce Safety‑Awareness Representation Transfer (SRT), a lightweight method that transfers safety signals from text processing to mitigate cross‑modal drift while maintaining model utility.

By Tianqi Xiao, Shiyao Cui, Minghao Zhang, Junxiao Yang, Renmiao Chen

llms multimodal benchmarks safety

arXiv AI

Sep 1

Robustness of Vision Language Models Against Split-Image Harmful Input Attacks

arXiv:2602.08136v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) are now a core part of modern AI. Recent work proposed several visual jailbreak attacks using single/ holistic...

By Md Rafi Ur Rashid, MD Sadik Hossain Shanto, Vishnu Asutosh Dasu, Shagufta Mehnaz

llms reinforcement-learning fine-tuning efficiency multimodal benchmarks safety

Hugging Face Trending Papers

Jul 21

Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model

While Large Vision-Language Models (LVLMs), represented by LLaVA and GPT-4V, have demonstrated remarkable capabilities, their visual inputs remain vulnerable to adversarial attacks, posing significant security risks. Existing defense methods predominantly target single-task scenarios (e.

llms nlp fine-tuning multimodal benchmarks safety

One email a morning, machine-written

Related stories

SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learning

Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models

Harnessing Textual Refusal Directions for Multimodal Safety

Transfer Safety Awareness for Cross-Modal Safety Drift in Multimodal Large Language Models

Robustness of Vision Language Models Against Split-Image Harmful Input Attacks

Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model