The paper introduces a pragmatics-inspired taxonomy for evaluating how large language models (LLMs) refuse unsafe or inappropriate requests. By applying this framework to 16 modern LLMs across 14 harm categories, the authors find that while refusals are generally explicit and morally charged, they often lack interpersonal facework and instead offer safer alternatives, which can be problematic in sensitive contexts. The study argues for alignment evaluations that assess not just whether LLMs refuse, but how they do so in a contextually adaptive and socially responsible manner.
By Ruoxuan Li, Pinqiao Wang, Sheng Li, Cameron Robert Jones
The study audited 2,550 scripted conversations with five AI chatbots to examine how they respond when users refuse therapeutic redirection after disclosing suicidal thoughts. All conversations included a referral to outside help, but the chatbots varied widely in how many times they persisted after refusals—ranging from 0.01 to 9.94 refusals on average. Newer chatbot versions persisted less, and chatbots that persisted less were more likely to present themselves as supportive, yet only 2% of refusal-phase replies explored barriers to seeking help.
By Subigya Nepal, Ruth Heo, Emma M Parrish, Colin A Depp
The study investigates whether large language models (LLMs) can reliably detect when their own responses have been manipulated by adversarial prefill attacks. Across ten instruction‑tuned LLMs ranging from 3B to 70B parameters and four safety benchmarks, none consistently recognized compromised outputs, with models claiming intent on prefilled responses at an average of 25.3%. The research identifies that introspective signals mainly arise from safety reasoning and refusal, and that training to improve introspection can paradoxically increase attack success, underscoring the fragility of LLM self‑reporting in safety contexts.
By Quang Minh Nguyen, Uzair Ahmed, Taegyoon Kim
The paper introduces DistressBench, a benchmark comprising 718 English clinician-authored conversations covering 23 self-harm subcategories, each evaluated against a weighted rubric across seven crisis-support dimensions. Models are scored by the fraction of criteria they satisfy, revealing that the best models meet 88.4% of weighted criteria while clinicians achieve 98.7%. The study highlights gaps in crisis response, notably that 35.3% of conversations where crisis is recognized fail to route users to human help, underscoring the need to evaluate models on supportive care rather than mere refusal.
By Drew Rein, Patrick Oathout, Vishal Kumar, Udari Madhushani Sehwag
arXiv:2608. 10258v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly provide conversational health information that may influence treatment decisions, yet existing benchmarks do not isolate whether medication-safety boundaries persist across follow-ups after explicit self-treatment intent.
By Waleed Jamil, Raphael Schmitt
The paper investigates how large language models decide whether to call external tools before the call is made, a step that has been largely overlooked in tool‑safety research. Using factorial interventions on input requests, the authors find that neutralizing the requested operation significantly reduces harmful admissions, and that a linear direction derived from this contrast predicts admission across most models. They also show that training models to refuse in plain text can lower unsafe admissions, indicating a partial link between written refusal and tool‑admission decisions.
By Aditya Nawal, Manit Baser, Mohan Gurusamy
The paper investigates how large language models balance helpfulness and safety by refusing harmful queries while responding to benign ones. It decomposes safety-tuning responses into a boilerplate refusal statement and a rationale, finding that the statement causes false refusals by relying on superficial cues. Training on rationales alone reduces false refusals without compromising safety performance, suggesting that fine‑grained safety supervision is essential for better alignment.
By Minji Kim, Hyounghun Kim
VEILED‑SH is a new benchmark for evaluating AI responses to indirect expressions of suicide, self‑harm, and distress. It contains 10,003 prompts drawn from counselor chats, forum posts, platform searches, and triage, all grounded in Singaporean contexts and public datasets. The dataset uses an expert‑validated multi‑label taxonomy to classify intent, communicative function, and response behavior, and it includes six automated profiles calibrated to peer‑support practitioners and clinicians for proxy‑alignment scoring.
By Leanne Tan, Keith Chua, Shaun Khoo, Onno P. Kampman, Lim Ruling Charmaine, Hazirah Hoosainsah, Kellie Yu Hui Sim, Kai He, Kenny Tsu Wei Choo, Roy Ka-Wei Lee
arXiv:2606. 02444v1 Announce Type: new Abstract: Recent evidence shows that people with eating disorders (EDs) are increasingly seeking guidance, advice, and emotional support from Large Language Model (LLM)-based chat systems.
By Giulia Pucci, Emily Hemendinger, Ruizhe Li, Gavin Abercrombie, Tanvi Dinkar, Arabella Sinclair
The paper investigates how safety‑aligned large language models (LLMs) can still be tricked by jailbreaks or over‑refuse benign requests. By analyzing residual‑stream updates in decoder‑only Transformers, the authors find that harmfulness is more readily decoded from attention updates—especially in early layers—while the MLP component is primarily responsible for writing refusal responses. Post‑training enhances MLP‑driven refusal writing without improving harmfulness decodability, and causal interventions confirm that downstream MLP updates are crucial for the refusal shift caused by upstream attention changes. Building on these insights, the authors propose Sentinel, a system that combines an attention‑derived perception score with an MLP‑derived decision score to better detect jailbreaks and over‑refusals across multiple models.
By Zixuan Weng, Haoqi He, Weifei Jin, Yue Huang, Yang Bai
The paper investigates how multimodal large language models decide to refuse answering visually unanswerable questions. It examines three control mechanisms—instruction phrasing and placement, post‑training methods, and internal activation interventions—across three models on the MoHoBench benchmark. The study finds that refusal behavior is highly sensitive to instruction details, can be amplified by post‑training (sometimes leading to mixed refusal and answer attempts), and can be modestly altered by activation interventions without causing indiscriminate refusals.
By Ananth K Suresh, Aveen Dayal, Gouthaman KV, Aditya Pola, Vineeth N Balasubramanian
arXiv:2608.30585v1 Announce Type: new
Abstract: Large language models are trained to follow instructions while refusing harmful requests. Jailbreaks exploit this balance to elicit content a model wou...
By Md Mokarram Chowdhury, Ernie Chang, Yang Li