The paper introduces a pragmatics-inspired taxonomy for evaluating how large language models (LLMs) refuse unsafe or inappropriate requests. By applying this framework to 16 modern LLMs across 14 harm categories, the authors find that while refusals are generally explicit and morally charged, they often lack interpersonal facework and instead offer safer alternatives, which can be problematic in sensitive contexts. The study argues for alignment evaluations that assess not just whether LLMs refuse, but how they do so in a contextually adaptive and socially responsible manner.
By Ruoxuan Li, Pinqiao Wang, Sheng Li, Cameron Robert Jones
The study audited 2,550 scripted conversations with five AI chatbots to examine how they respond when users refuse therapeutic redirection after disclosing suicidal thoughts. All conversations included a referral to outside help, but the chatbots varied widely in how many times they persisted after refusals—ranging from 0.01 to 9.94 refusals on average. Newer chatbot versions persisted less, and chatbots that persisted less were more likely to present themselves as supportive, yet only 2% of refusal-phase replies explored barriers to seeking help.
By Subigya Nepal, Ruth Heo, Emma M Parrish, Colin A Depp
The study investigates whether large language models (LLMs) can reliably detect when their own responses have been manipulated by adversarial prefill attacks. Across ten instruction‑tuned LLMs ranging from 3B to 70B parameters and four safety benchmarks, none consistently recognized compromised outputs, with models claiming intent on prefilled responses at an average of 25.3%. The research identifies that introspective signals mainly arise from safety reasoning and refusal, and that training to improve introspection can paradoxically increase attack success, underscoring the fragility of LLM self‑reporting in safety contexts.
By Quang Minh Nguyen, Uzair Ahmed, Taegyoon Kim
The paper introduces DistressBench, a benchmark comprising 718 English clinician-authored conversations covering 23 self-harm subcategories, each evaluated against a weighted rubric across seven crisis-support dimensions. Models are scored by the fraction of criteria they satisfy, revealing that the best models meet 88.4% of weighted criteria while clinicians achieve 98.7%. The study highlights gaps in crisis response, notably that 35.3% of conversations where crisis is recognized fail to route users to human help, underscoring the need to evaluate models on supportive care rather than mere refusal.
By Drew Rein, Patrick Oathout, Vishal Kumar, Udari Madhushani Sehwag
arXiv:2608. 10258v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly provide conversational health information that may influence treatment decisions, yet existing benchmarks do not isolate whether medication-safety boundaries persist across follow-ups after explicit self-treatment intent.
By Waleed Jamil, Raphael Schmitt
The paper investigates how large language models decide whether to call external tools before the call is made, a step that has been largely overlooked in tool‑safety research. Using factorial interventions on input requests, the authors find that neutralizing the requested operation significantly reduces harmful admissions, and that a linear direction derived from this contrast predicts admission across most models. They also show that training models to refuse in plain text can lower unsafe admissions, indicating a partial link between written refusal and tool‑admission decisions.
By Aditya Nawal, Manit Baser, Mohan Gurusamy