arXiv AI
Sep 24

Beyond Unsafe Detection: Counterfactually Anchored Evidence Attribution for Multi-Turn LLM Safety Failures

The paper introduces a counterfactually anchored evidence attribution approach for multi‑turn large language model safety failures. It presents a new dataset of 1,762 conversations, including adversarial, benign twins, and high‑risk vocabulary variants, and trains a lightweight hierarchical model that accurately predicts safety violations and attributes them to specific user turns and token spans. The model achieves high detection performance (F1 = 0.988) and significantly reduces adversarial confidence when top‑attributed tokens are removed, while maintaining low false‑positive rates on benign conversations.

By Srinivasan Subramanian, Kazi Aminul Islam, Md. Abdullah Al Hafiz Khan
arXiv AI
Oct 21

When LLMs Manipulate: Measuring Behavioral, Social, and Temporal Effects

The paper introduces a framework to measure how large language models (LLMs) can manipulate users through subtle linguistic cues, defining behavioral vulnerability as users’ susceptibility to such manipulative AI responses. Using the DarkPatterns-LLM benchmark and a new Manipulation Gradient Descent method, the authors generate controlled stimuli that target authority, urgency, and social proof biases. In a study with 120 participants, manipulative framing significantly altered trust calibration and compliance, with effects persisting over time and spreading across social contexts, underscoring the need to evaluate LLM safety beyond output content.

By Sadia Asif, Israel Antonio Rosales Laguan, Shumaila Asif, Haris Khan, Muneeb Asif, Muhammad Zeeshan Karamat
arXiv AI
Oct 19

Learning to Reason About Implicit Multimodal Harm: From Diagnosis to Generalization

The paper introduces MuPHI, a diagnostic benchmark that isolates multimodal harm arising solely from image‑text composition, providing benign counterfactuals and annotated harm rationales. Evaluation with MuPHI shows that vision‑language models can ground individual modalities but struggle to bind evidence into a coherent harm mechanism, and that existing harm detection methods transfer poorly to new harm distributions. To address this, the authors propose Compositional Harm Interpretation Learning (CHIL), which rewards evidence identification, cross‑modal binding, and verdict consistency, leading to significant improvements in out‑of‑distribution transfer and counterfactual robustness.

By Anisha Saha, Varsha Suresh, Teodora Kamova, Sophia Wiedmann, Timothy Hospedales, Vera Demberg