arXiv Computation and Language

Language Shapes Instruction Hierarchy Compliance in Multilingual LLMs

The paper introduces XIH-Bench, a multilingual benchmark for evaluating instruction hierarchy (IH) compliance in large language models across six languages, four domains, and three IH settings. It finds that IH compliance varies with language, showing an asymmetry where a language can strengthen compliance in a higher-priority role but disrupt it in a lower-priority role, and that cross-language conflicts yield higher compliance than same-language conflicts—a phenomenon called the Language Boundary Effect. The study also highlights that language specialization can make lower-priority instructions in favored languages harder to override, raising potential reliability and security concerns.

arXiv AI
3d ago

Global Security Blind Spots: Translation as a Safety Bypass in Low-Resource Languages

The paper investigates how large language models (LLMs) can be bypassed by translating harmful requests from English into low‑resource languages. By testing ten popular LLMs across 79 languages, the authors find that safe English completions often fail to remain safe after translation, with bypass rates between 21.9% and 44.4%. The risk is unevenly distributed across language families and varies significantly between models, making it hard to rank models by safety consistently.

By Jalil Alva, David Harper, Abigail E. Thornton, Rafael S. Villalobos Alva, Leonardo Kenji Minemura Suazo, Konstantinos Karageorgos, Diego Velazquez, Sergio A. Serrano, Marvin Vargas Flores, Aaron Schliem, Fernando F. Migone
arXiv AI
Oct 16

Multi-SFT: Permissively-Licensed Multilingual Instruction-Following

Multi‑SFT is a new, commercially permissive dataset comprising 502,818 instruction‑response pairs in five European languages, created through translation and regeneration pipelines to address gaps where simple translation is insufficient. Human evaluation shows strong inter‑annotator agreement (≥0.87) across all languages. Fine‑tuning nine models from three families on Multi‑SFT yields performance competitive with larger, proprietary mixes and official instruction‑tuned releases, especially on safety‑related benchmarks.

By Daniela Baiamonte, Kamyar Zeinalipour, Elena Fano, Stefano Simonazzi, Andrea Zugarini
arXiv AI
Sep 10

Many-Tier Instruction Hierarchy in LLM Agents

The paper introduces Many-Tier Instruction Hierarchy (ManyIH), a new framework for resolving conflicts among instructions with arbitrarily many privilege levels in large language model agents. It presents ManyIH-Bench, a benchmark featuring 853 agentic tasks that require navigating up to 12 levels of conflicting instructions across 46 real-world agents. Experiments show current models achieve only about 40% accuracy when instruction conflict scales, highlighting a gap in fine-grained, scalable conflict resolution.

By Jingyu Zhang, Tianjian Li, William Jurayj, Hongyuan Zhan, Benjamin Van Durme, Daniel Khashabi
arXiv AI
Oct 16

Language Is a Security Surface for Agent Guards

The study investigates whether the language of a user’s instruction affects the safety verdicts of agent guards. By translating 1,220 labeled steps from an English benchmark into German, French, Turkish, Swahili, and Amharic while keeping all other data identical, the authors find that many open, small, and purpose‑built guards alter their verdicts when the instruction is translated, with larger effects for rarer languages. The impact varies with guard size, with smaller models showing greater loss of recall and larger models (e.g., Gemini 3.8 Flash) remaining largely unchanged.

By Hasan Demirkiran, Jens Ernstberger
arXiv AI
Aug 18

LLM Safety Alignment in Low-Resource Languages: A Systematic Literature Review

arXiv:2608. 14626v1 Announce Type: cross Abstract: Large Language Models (LLMs) have achieved substantial progress in safety alignment, yet their safety guarantees remain significantly weaker in low-resource and multilingual settings than in high-resource languages.

By Valdini Douglace Lemofouet, Blessing Ngozi Uzor, Paula Chikaodinaka Anyanwu, Danielle Blanche Kapsa, Sukairaj Hafiz Imam, P Sam Sahil, Abigail Oppong, Tassallah Abdullahi, Clemencia Siro, Idris Abdulmumin, Seid Muhie Yimam, Shamsuddeen Hassan Muhammad