The study compares an English-only and a bilingual decoder-only model, each 310 M parameters, trained on eight diverse languages while controlling for English exposure, compute, and document overlap. After aligning on shared English vocabulary, the authors find that token embeddings appear similar, but the deeper hidden states used for prediction diverge across models. This hidden‑state mismatch grows through middle transformer layers and persists despite controls, indicating that contextual processing differs between the models.
"whyItMatters":"The findings show that embedding alignment can conceal significant internal representation differences, which is crucial for any downstream work that assumes aligned multilingual models are interchangeable."
By Anjishnu Mukherjee, Ziwei Zhu, Antonios Anastasopoulos
The paper investigates why large language models sometimes hallucinate when asked about facts in a language different from the one in which the facts were learned. By training small Transformer models on synthetic multilingual datasets, the authors show that the degree of correlation between facts and their learning language (informativeness) and the ease of language identification (extractability) determine whether models develop unified or separate representations across languages. Unified representations enable cross‑lingual fact transfer, while separate representations do not. The study proposes a unifying perspective on cross‑lingual transfer and suggests training methods to promote representational unification.
By Carter Blum, Katja Filippova, Ann Yuan, Asma Ghandeharioun, Julian Zimmert, Fred Zhang, Jessica Hoffmann, Tal Linzen, Martin Wattenberg, Lucas Dixon, Mor Geva
arXiv:2608.03446v2 Announce Type: replace
Abstract: Multilingual large language models (LLMs) have been shown to perform better on non-English classification tasks when the representations of the giv...
By Adnan Al Ali, Kathy H\"ammerl, Jind\v{r}ich Libovick\'y, Alexander Fraser
The paper investigates how to fairly compare language models across languages, noting that current evaluation methods vary widely and lack empirical validation. By training controlled monolingual models on parallel data and testing multilingual LLMs, the authors find that many normalized metrics suffer from biases due to tokenization, encoding, and orthographic differences. Instead, they recommend using sentence‑level negative log‑likelihood over semantically equivalent sequences for more reliable cross‑lingual comparisons.
By Xiulin Yang, Ethan Gotlieb Wilcox, Catherine Arnett
The study investigates whether multilingual language models solve mathematical problems using shared or language‑specific internal representations. By replaying reasoning traces across six languages and applying Centered Kernel Alignment, the authors identify layers with cross‑language alignment and train two sparse autoencoders— a baseline and a new Geometry‑Invariant SAE (GI‑SAE) that adds an InfoNCE loss to encourage language‑agnostic activations. Functional interchangeability tests reveal that higher geometric similarity does not always translate to interchangeable features, and cross‑language sharing varies by model and depth, with GI‑SAE amplifying existing structure in some models but not others.
By Igor Bogdanov, Changcheng Huang
arXiv:2605.31171v2 Announce Type: replace-cross
Abstract: Multilingual Information Retrieval (MLIR) reflects real-world search environments in which queries and relevant documents may appear in diffe...
By Youngjoon Jang, Seongtae Hong, Heuiseok Lim