The paper examines how automated safety systems unevenly flag harmful content, disproportionately labeling African American English as offensive and misclassifying content from drag queens and white nationalists. It identifies seven failure modes rooted in vocabulary-based harm detection and proposes an evaluation protocol that highlights gaps in performance across languages. Applied to five open systems in eleven languages, the study finds that none achieve native-speaker accuracy, and higher capability often increases discrimination, widening the disparity between best- and worst-served language communities.
By Gabriel Lars Sabadin, Dr Nicola Harding, Sarah Kingston
The paper introduces GUISE, a benchmark of criminal‑assistance scenarios drawn from real‑world cases and organized using the UNODC International Classification of Crime for Statistical Purposes. It evaluates large language models under three linguistic framings and across five cultures, finding that framing and cultural adaptation significantly increase unsafe compliance rates. The study also shows that unsafe responses from the least‑safe models exhibit strong Anglocentric language, with leakage growing alongside harmfulness.
By Mohsinul Kabir, Md. Tariquzzaman, Karam S. R. Elradie, Polydoros Giannouris, Yuechen Jiang, Shaoxiong Ji, Sophia Ananiadou
arXiv:2605. 17173v2 Announce Type: replace-cross Abstract: Large language models exhibit safety degradation in non-English languages.
By Max Zhang, Ameen Patel, Sang T. Truong, Sanmi Koyejo
arXiv:2610.14443v1 Announce Type: cross
Abstract: Model cards, system cards, and safety reports increasingly document LLM safety behaviors, yet safety evaluations typically start from predefined benc...
By Kausar Moshood, Jace Bolante, Roshan Aditya Murali, Manish Motwani
arXiv:2608. 14577v1 Announce Type: cross Abstract: Frontier large language models (LLMs) safety evaluation has largely treated harmful generation as an attack outcome rather than as an object of analysis.
By Zhouyuan Ma, Yutao Wu, Hanxun Huang, Xiang Zheng, Xiao Liu, Yixin Cao, Zuxuan Wu, Xingjun Ma, Yu-Gang Jiang
arXiv:2607. 15861v1 Announce Type: cross Abstract: Moderation systems increasingly rely on external toxicity tools, but those tools are unreliable under code-mixing, transliteration, slang, and language mismatch.
By Indraveni Chebolu, Rohan Singh, Arnab Mallick, Harmesh Rana
arXiv:2608. 13695v1 Announce Type: cross Abstract: Large language model providers routinely cite multilingual safety benchmarks spanning a dozen or more languages as evidence that their models are safe for non-English-speaking users.
By Chialuka Prisca-Mary Onuoha, Bright Etornam Sunu, Rashidat Sikiru
Retrieval‑augmented generation (RAG) can expose language models to misleading evidence. In a study of 200 SciFact questions, the Gemma‑3‑12b model’s ability to detect poisoned claims dropped from 25% when a single poisoned document was present to 10% when six mutually agreeing poisoned documents dominated the context. The majority condition caused 34 of 38 questions to shift from detected to undetected, indicating a conformity‑like bias.
By Mehrdad Ghassabi, Audrina Ebrahimi, Zahra Zojaji, Amin Milani Fard
arXiv:2611.02522v1 Announce Type: new
Abstract: Programs that use a language model as a classifier act on its most probable label. Text injected into the input can move that label to one the attacker...
By Junwei Yao, Jianan Liu, Yongxi Zhou, Zike Yang
arXiv:2609.13579v1 Announce Type: new
Abstract: Safety research often focuses on model-generated harms, but users may also direct hostility, coercion, and adversarial pressure at models. Understandin...
By Fanqi Zeng, Sadid A. Hasan, Chaocheng He
The paper introduces DiagSafe, a diagnostic evaluation framework that replaces binary safe/unsafe labels with prompt-level rubrics grounded in AI safety policies. DiagSafe uses a bilingual benchmark with an 84‑label safety taxonomy, comprising DiagBase and DiagAttack subsets, to assess safety and usefulness independently. Experiments show DiagSafe achieves high agreement with human annotations (95.34%) and strong inter‑judge consistency (Cohen’s κ = 0.864), revealing significant differences in unsafe rates and vulnerabilities across leading LLMs, especially in Chinese and multi‑turn contexts.
By Hongyu Li, Xinlong Huang, Zhan Yu, Jing Zhao, Zhe Zhao, Juntao Zhong, Xiong Kuang, Rui Yuan, ZhenXiang Yan, Huilin Xu, Haitao Yu, Lilin Wang, Rui Chen, Blaze Chen, Fengzong Lian, Renfen Hu
The paper investigates whether large language models’ refusal behavior in text transfers to tool‑calling contexts. A benchmark of 2,304 prompts across four domains compares text‑only, standard tool‑enabled, and adversarially framed tool‑enabled settings, revealing that 10–52 % of harmful requests refused in chat still lead to unsafe tool calls. The authors identify a refusal‑associated direction in the models’ residual stream, showing that lower activation along this direction predicts unsafe calls, but restoring the signal only partially mitigates the issue and often suppresses benign tool use.
By Ekagra Gupta, Raj Saha, Aniekan-Abasi Inyang, Adam Al Gharib