The paper introduces a trace-level divergence framework for evaluating multilingual large language models, breaking down agreement into structural similarity, order consistency, and holistic similarity. Experiments across three reasoning benchmarks, seven languages, and six leading LLMs show that conventional holistic similarity metrics overstate agreement, with up to 67.3% of top-scoring examples lacking structural alignment. Human judgments align more closely with structural similarity, highlighting language-specific divergence and the need for trace-level evaluation in multilingual deployments.
arXiv:2607. 06327v1 Announce Type: cross Abstract: Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English.
arXiv:2608. 10444v1 Announce Type: cross Abstract: Large language models (LLMs) have made substantial progress on reasoning tasks that require increasingly long and complex inferential chains.
arXiv:2604.04720v2 Announce Type: replace Abstract: Large Reasoning Models (LRMs) still exhibit large performance gaps between English and other languages, yet much current work assumes these gaps ca...
The study investigates the performance gap between native-language reasoning and English-pivoted reasoning in large language models. By creating extensive multilingual reasoning datasets and fine‑tuning specialists on Qwen/Qwen3-8B-Base, the authors find that the native reasoning gap is much smaller (1.9–3.5%) than previously reported. They analyze weight‑space changes, discover a language‑agnostic reasoning core in the middle layers, and propose a Layer Swap technique that transfers these mid‑layer updates from an English specialist to native specialists, effectively closing most of the gap while maintaining native chain‑of‑thought output.
arXiv:2608. 08447v1 Announce Type: cross Abstract: Multilingual reasoning models are commonly evaluated by whether they arrive at the correct answer, but not by whether they preserve the intended language while reasoning and responding.
XHotpotQA is a new benchmark for cross‑lingual knowledge composition in multi‑hop question answering. It presents each instance as an evidence‑dependency graph with explicit language assignments for the question, bridge evidence, answer‑bearing evidence, and distractors, and includes 15,661 training and 7,405 validation examples with sentence‑level support supervision. The dataset reveals significant performance drops when evidence spans language boundaries, providing a diagnostic tool for systems that must integrate evidence across languages.
The study explores how the language used for reasoning affects retrieval‑augmented generation (RAG) in a monolingual German setting. Using a German RAG question‑answering testbed based on the tabletop game The Dark Eye, the authors show that aligning the reasoning language with the query and retrieved documents improves performance, with German reasoning outperforming French reasoning. However, German reasoning still does not surpass the model’s native English reasoning, indicating that native multilingual reasoning is necessary for optimal results.
arXiv:2609.37104v1 Announce Type: cross Abstract: Open-source reasoning models provide unequal access to reasoning capability across languages. When a model can solve a problem but cannot deliver a c...
arXiv:2601. 14063v2 Announce Type: replace-cross Abstract: Cross-cultural competence in large language models (LLMs) requires understanding and adapting Culture-Specific Items (CSIs) across varying cultural contexts.
The paper introduces a knowledge‑graph‑based evaluation framework, S3KG, to assess whether large language models truly understand context in question answering tasks. S3KG combines structural and semantic signals into a single similarity score and is paired with a diagnostic analysis that pinpoints reasoning errors at the triplet level. Across nine benchmarks, the method outperforms existing baselines, achieving up to +7.6 F1 points and an AUROC of 0.973.
The paper investigates whether language models can reason across languages by introducing a two‑hop question answering task that requires inference over two multilingual documents. Results show that models are more sensitive to language variation in answer‑span documents than in bridging documents, and that up to 33% of multilingual cases involve correct final answers despite failing to infer bridging information in the first step. The study also reveals an 18% composition failure rate and proposes a three‑stage SUBQ prompting method that improves accuracy from 10.1% to 66.5%.
arXiv:2607. 05992v1 Announce Type: cross Abstract: Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites.