The paper investigates attention heads in multilingual Transformer models, distinguishing between retrieval heads that pull information from context and a newly identified class called Retrieval‑Transition heads (RTH) that direct the model toward a specific target language. Experiments across four multilingual benchmarks and two model families show that masking RTHs causes a larger performance drop than masking retrieval heads, indicating RTHs are crucial for chain‑of‑thought reasoning in multilingual LLMs. The study thus clarifies which attention heads are responsible for mapping to target languages, advancing our understanding of multilingual language models.
By Shaswat Patel, Vishvesh Trivedi, Yue Han, Yihuai Hong, Eunsol Choi
arXiv:2608. 15964v1 Announce Type: cross Abstract: Language-specific competency (LSC) is the phenomenon of a language model performing better or worse depending on the language of the prompt.
By Ishika Agarwal, Arkajyoti Charaborty, Tanner Sorensen, Neha Gupta, Andreas Stolcke
The study investigates the performance gap between native-language reasoning and English-pivoted reasoning in large language models. By creating extensive multilingual reasoning datasets and fine‑tuning specialists on Qwen/Qwen3-8B-Base, the authors find that the native reasoning gap is much smaller (1.9–3.5%) than previously reported. They analyze weight‑space changes, discover a language‑agnostic reasoning core in the middle layers, and propose a Layer Swap technique that transfers these mid‑layer updates from an English specialist to native specialists, effectively closing most of the gap while maintaining native chain‑of‑thought output.
By Maxence Lasbordes, Am\'elie Chatelain, Djam\'e Seddah
arXiv:2410.02343v2 Announce Type: replace
Abstract: Large language models (LLMs) routinely fail to output the correct option in multiple-choice question answering (MCQA) while encoding the answer int...
By Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Anastasia Voznyuk, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, Serguei Barannikov
The paper introduces “Rel-Bloom,” a multilingual diagnostic benchmark of 450 multiple-choice questions that tests language models’ ability to reason over relational facts supplied directly in prompts. It evaluates 43 models across six languages and two multilingual settings, revealing that accuracy drops sharply from Apply to Analyze levels, that Evaluate items remain a bottleneck even for high-performing models, and that difficulty rankings are consistent across languages. The study also finds that counterfactual contexts are generally harder than factual ones, and that removing entity identities can sometimes improve higher-level performance.
By Zohreh Abbasi Liasi, Batoul Hammoud, Anas Madkoor, Ehsaneddin Asgari
The study explores how the language used for reasoning affects retrieval‑augmented generation (RAG) in a monolingual German setting. Using a German RAG question‑answering testbed based on the tabletop game The Dark Eye, the authors show that aligning the reasoning language with the query and retrieved documents improves performance, with German reasoning outperforming French reasoning. However, German reasoning still does not surpass the model’s native English reasoning, indicating that native multilingual reasoning is necessary for optimal results.
By Oliver Hauck, Mario Sanz-Guerrero, Katharina von der Wense
arXiv:2609.37104v1 Announce Type: cross
Abstract: Open-source reasoning models provide unequal access to reasoning capability across languages. When a model can solve a problem but cannot deliver a c...
By Hongyang Li, Xiao Li, Caesar Wu, Gr\'egoire Danoy, Pascal Bouvry
arXiv:2607. 01002v1 Announce Type: cross Abstract: In long-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy-pasting them.
By Aryo Pradipta Gema, Beatrice Alex, Pasquale Minervini
The paper "Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparities" introduces the Waldo benchmark, a multilingual QA dataset built from Wikipedia that focuses on knowledge gaps and conflicts across languages. It evaluates eight models in five languages and finds that when a fact is missing in one language, models tend to use evidence from the other language, but when conflicting accounts exist, responses align strongly with the query language, leading to different answers for semantically identical questions. The study also explores mitigation strategies, including ablating attention heads and LoRA-based training, which can reduce the preference gap by up to 61.5%.
By Dayeon Ki, Ruochen Zhang, Silviu Cucerzan, Ryen W. White, Ning Gao
arXiv:2606. 11198v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) systems inject external knowledge to improve LLM outputs, yet the format of injected content -- distinct from its semantic relevance -- can independently distort the model's attention distribution.
By Yuqi Zhang, Di Zhang
The paper introduces CLLPU, a multilingual benchmark for evaluating how well large language models can unlearn specific knowledge while controlling its propagation across languages. CLLPU defines two forgetting scenarios—common-goal forgetting, which requires suppression across all languages, and language-conditioned forgetting, which limits suppression to a single language. Using 800 knowledge-unit pairs and 72,000 QA instances in ten languages, the authors test six methods on Llama‑3.1‑8B‑Instruct and find that universal suppression often fails, while language‑specific suppression can unintentionally spread to other languages, highlighting the difficulty of propagation control in multilingual unlearning.
By Pengyang Shao, Chuanpeng Lu, Wei Qin, Yanzheng Jin, Xiaohao Liu, Xi Ai, Kenji Kawaguchi, Richang Hong
The paper introduces Jev, a specialized decision model that selects among given options rather than generating text, and benchmarks it against 19 large language models (LLMs) across 13 multiple-choice tasks in knowledge, reasoning, and multilingual domains. Jev performs competitively with frontier LLMs on knowledge and commonsense benchmarks, achieving the top score on MMLU-Redux and ARC-Challenge, and surpasses most representative and all small LLMs except in mathematical word problems where it lags behind. The study demonstrates that a specialized decision model can match general-purpose LLMs on knowledge-based decisions but struggles with multi-step calculations.
By Xing Li, Qingcheng Chang, Jinzhong Ning, Changfeng Xu, Shenlong Zhang, Yijia Zhang, Ling Luo, Hongfei Lin