arXiv AI
Aug 12
Why Do Safety Guardrails Degrade Across Languages?
arXiv:2605. 17173v2 Announce Type: replace-cross Abstract: Large language models exhibit safety degradation in non-English languages.
Red-Teaming Large Language Models in Low-Resource African Languages: A Case Study of Yoruba and Nigerian Pidgin
The study evaluates six state‑of‑the‑art large language models on 400 adversarial prompts in Yoruba and Nigerian Pidgin, categorised into four harm types. Native‑speaker annotators scored responses and assessed reliability with Cohen’s κ, revealing that many models failed to understand the prompts, leading to low rejection rates in the African languages. Harmful content was frequently produced, with attack success rates higher in Nigerian languages and significant cross‑lingual safety gaps, especially for GPT‑4o Mini and Aya 101.
Related stories
arXiv Computation and Language
Oct 26
AfriGuard: Safety and Robust Refusal For African Languages
arXiv:2610.26474v2 Announce Type: replace Abstract: Large language models (LLMs) designed to interact with users are typically trained to respond appropriately to harmful or toxic content. However, e...
arXiv AI
Oct 15
XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity
arXiv:2605.05662v2 Announce Type: replace-cross Abstract: Current LLM safety benchmarks are predominantly English-centric and often rely on translation, failing to capture country-specific harms. Mor...
arXiv AI
Sep 4
IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks
IndicSafeEval is a new evaluation framework that tests the safety robustness of large language models against persuasion-based jailbreak attacks in Indian languages. The benchmark includes 7,200 adversarial prompts covering ten safety-critical content categories, six persuasive strategies, and four languages (Hindi, Bengali, Marathi, Punjabi). Experiments show that model safety varies significantly across languages, prompt styles, and risk categories, revealing that current English-centric safety evaluations miss important multilingual vulnerabilities.
Hugging Face Trending Papers
Jun 8
Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis
Multilingual safety evaluation of large language models (LLMs) has predominantly relied on direct translation (DT) of English benchmarks into target languages - an approach that converts surface-level linguistic form while failing to reflect the cultural context embedded in threat scenarios, social norms, and legal frameworks. We construct paired DT and culturally-adapted (CA) datasets via 1:1 seed matching for four languages - Korean (KO), Japanese (JA), Thai (TH), and Khmer (KM) - and compare Attack Success Rate (ASR) and Cultural Realism scores across four open-source LLM.
arXiv AI
Jun 2
TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages
arXiv:2606. 01322v1 Announce Type: cross Abstract: Safety evaluation of Large Language Models (LLMs) remains heavily English-centric, leaving Low-Resource Languages (LRLs), particularly African ones, critically underexplored.
Hugging Face Trending Papers
Aug 11
The Illusion of Cross-Lingual Safety in Low-Resource Languages
Safety alignment in large language models (LLMs) is largely developed in English, assuming these safeguards generalize across multilingual settings. However, this assumption remains underexplored and exposes a vulnerability in low-resource languages.
Hugging Face Trending Papers
Sep 3
IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks
IndicSafeEval is a new evaluation framework that tests the safety robustness of large language models against persuasion-based jailbreak attacks in Indian languages. The benchmark covers ten safety-critical content categories, six persuasive strategies, and four languages—Hindi, Bengali, Marathi, and Punjabi—producing 7,200 adversarial prompts. Experiments show that model safety varies significantly across languages, prompt styles, and risk categories, highlighting gaps in current English-centric safety assessments.
arXiv AI
Oct 28
Aligning Large Language Models for Safety in African Languages: A Case Study of Hausa
The paper introduces HausaSafeLM, a safety preference dataset for Hausa comprising 10,000 prompts (3,000 human‑corrected and 7,000 machine‑translated) across six harm categories. Using Qwen2.5‑7B‑Instruct, the authors compare supervised fine‑tuning (SFT) and SFT followed by Direct Preference Optimization (DPO) on 500 harmful and 200 benign Hausa prompts, finding that SFT improves refusal fluency and precision—especially with larger, noisier data—while DPO sharpens precision at the cost of linguistic quality. No single approach simultaneously maximizes safety, helpfulness, and fluency, and the study shows that refusal‑rate metrics alone are unreliable indicators of safety in low‑resource languages.
arXiv AI
Jun 9
Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis
arXiv:2606. 09178v1 Announce Type: cross Abstract: Multilingual safety evaluation of large language models (LLMs) has predominantly relied on direct translation (DT) of English benchmarks into target languages - an approach that converts surface-level linguistic form while failing to reflect the cultural context embedded in threat scenarios, social norms, and legal frameworks.
arXiv AI
Jul 8
ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety
arXiv:2605. 14152v2 Announce Type: replace-cross Abstract: Safety evaluations for large language models (LLMs) increasingly target high-stakes National Security and Public Safety (NSPS) risks, yet multilingual safety is mostly assessed through translation-only benchmarks that preserve the underlying scenario, leaving how language and geopolitical context interact largely unexamined beyond a few language pairs.
arXiv Computation and Language
Oct 23
Harm Is Behavioural, Moderation Is Lexical: How Language and Culture Decide Whom Automated Safety Systems Protect and Whom They Silence
The paper examines how automated safety systems unevenly flag harmful content, disproportionately labeling African American English as offensive and misclassifying content from drag queens and white nationalists. It identifies seven failure modes rooted in vocabulary-based harm detection and proposes an evaluation protocol that highlights gaps in performance across languages. Applied to five open systems in eleven languages, the study finds that none achieve native-speaker accuracy, and higher capability often increases discrimination, widening the disparity between best- and worst-served language communities.