The study investigates whether monolingual language models, trained without joint multilingual objectives, develop cross-lingual alignment. By evaluating models such as Goldfish and independently built monolingual systems, the authors find that alignable representational geometry emerges across layers, strengthening with larger data, larger models, or closer linguistic proximity. A single Procrustes rotation on parallel sentences can map hidden states between models, and applying this rotation to a German model’s residuals swaps factual predictions to those of the donor English model, demonstrating functional transfer.
By Ej Zhou, Suchir Salhan, Catherine Arnett, Anna Korhonen
Large language models show strong cross‑lingual abilities, but previous studies have examined them only at isolated training points. This work pretrains a 1.7B multilingual model on nine languages, recording checkpoints at fine granularity to track translation development. Using word‑level translation and a new dataset, the authors find that early learning is dominated by token‑level copying and surface similarities, while later stages involve more general translation mechanisms and refined copying.
By Felicia K\"orner, Maria Matveev, Florian Eichin, Gitta Kutyniok, Barbara Plank, Michael A. Hedderich
The paper challenges the assumption that improving cross‑lingual alignment automatically enhances cross‑lingual transfer. Using XLM‑R models aligned on token, sentence, and masked‑language‑modeling objectives across four language pairs, the authors evaluate zero‑shot transfer on part‑of‑speech tagging and sentence classification. They find that embedding‑based alignment metrics poorly predict downstream performance and that alignment and task gradients are often nearly orthogonal, especially when operating at different representational levels.
By Yana Veitsman, Yihong Liu, Hinrich Sch\"utze
The paper investigates weight‑space merging of independently fine‑tuned multilingual machine translation models. Experiments show that merging is more successful when models share a target language, yet it still cannot match the peak performance of language‑specific checkpoints. When target languages differ, performance drops sharply, and analysis reveals that overlapping neuron activation and incompatible upper‑layer geometries cause these failures.
By Baban Gain, Trilok Nath Singh, Asif Ekbal
The paper investigates why large language models (LLMs) fail to transfer knowledge across languages. By pretraining 360M- and 7B-parameter models, the authors show that poor cross‑lingual generalization arises during pretraining and persists despite standard fixes. Using a controlled bilingual setup with identical text but disjoint token spaces, they demonstrate that token disjointness alone causes knowledge compartmentalization, and that mapping languages into a shared token space via word‑wise translation markedly improves cross‑lingual performance, recovering up to 12.6% of native‑language learning efficiency.
By Adam Gaber, Uriel Dolev, Elisabeth Fittschen, Bobby Cheng, Yuval Marton, Leshem Choshen
The study evaluates how different input representations—orthographic text, IPA transcription, and romanization—affect cross‑lingual transfer in autoregressive multilingual language models. Across three model sizes and eight languages grouped into typologically motivated pairs, romanized pretraining consistently outperforms native orthography and IPA, especially as model scale increases. Fine‑tuning a text‑pretrained model on romanized data can harm performance on languages already covered by the base model, suggesting romanization should be integrated at pretraining rather than applied later.
By Muge Zhang, Aaron Jencks, Krishna Badikela, Yulia Tsvetkov, Sachin Kumar
The paper investigates how to fairly compare language models across languages, noting that current evaluation methods vary widely and lack empirical validation. By training controlled monolingual models on parallel data and testing multilingual LLMs, the authors find that many normalized metrics suffer from biases due to tokenization, encoding, and orthographic differences. Instead, they recommend using sentence‑level negative log‑likelihood over semantically equivalent sequences for more reliable cross‑lingual comparisons.
By Xiulin Yang, Ethan Gotlieb Wilcox, Catherine Arnett
arXiv:2601.16390v2 Announce Type: replace-cross
Abstract: Large language models exhibit strong multilingual capabilities, yet significant performance gaps persist between dominant and non-dominant la...
By Rhitabrat Pokharel, Ameeta Agrawal, Tanay Nagar
arXiv:2610.07168v1 Announce Type: new
Abstract: Representations of translated sentences are similar in the inner layers of multilingual language models -- an observation connected to the platonic rep...
By Darshil Doshi, Wenjie Zhou, Corinna Elena Wegner, Daniel J. Korchinski, Santiago Acevedo, Matthieu Wyart
The paper evaluates four metrics—CKA, ANC, GMM dominance per token, and ILO—used to measure cross‑lingual representation sharing in multilingual language models. Across 21 models ranging from 125 M to 14 B parameters, the metrics disagree, and the authors attribute this to anisotropy, where representations cluster in a narrow embedding cone. Only ILO shows a strong, robust correlation with cross‑lingual transfer performance (Spearman’s ρ = 0.90) after controlling for model size, family, and task variation, leading the authors to recommend ILO as the primary metric alongside anisotropy diagnostics.
By Oskar Holmstr\"om, Marcel Bollmann, Marco Kuhlmann
The paper investigates why large language models sometimes hallucinate when asked about facts in a language different from the one in which the facts were learned. By training small Transformer models on synthetic multilingual datasets, the authors show that the degree of correlation between facts and their learning language (informativeness) and the ease of language identification (extractability) determine whether models develop unified or separate representations across languages. Unified representations enable cross‑lingual fact transfer, while separate representations do not. The study proposes a unifying perspective on cross‑lingual transfer and suggests training methods to promote representational unification.
By Carter Blum, Katja Filippova, Ann Yuan, Asma Ghandeharioun, Julian Zimmert, Fred Zhang, Jessica Hoffmann, Tal Linzen, Martin Wattenberg, Lucas Dixon, Mor Geva
arXiv:2608.03446v2 Announce Type: replace
Abstract: Multilingual large language models (LLMs) have been shown to perform better on non-English classification tasks when the representations of the giv...
By Adnan Al Ali, Kathy H\"ammerl, Jind\v{r}ich Libovick\'y, Alexander Fraser