The study probes Gemma‑2‑9B‑IT with Sparse Autoencoders across English, Hebrew, and Russian to examine how multilingual LLMs handle informal register. By using a dataset of polysemous terms that appear in literal and informal contexts, the authors isolate pragmatic register processing from lexical cues. They discover a small, robust cross‑linguistic core that forms an informal register subspace, which becomes clearer in deeper layers and can causally shift output formality across all tested languages, even transferring zero‑shot to six unseen languages.
By Uri Z. Kialy, Avi Shtarkberg, Ayal Klein
The study evaluates how different input representations—orthographic text, IPA transcription, and romanization—affect cross‑lingual transfer in autoregressive multilingual language models. Across three model sizes and eight languages grouped into typologically motivated pairs, romanized pretraining consistently outperforms native orthography and IPA, especially as model scale increases. Fine‑tuning a text‑pretrained model on romanized data can harm performance on languages already covered by the base model, suggesting romanization should be integrated at pretraining rather than applied later.
By Muge Zhang, Aaron Jencks, Krishna Badikela, Yulia Tsvetkov, Sachin Kumar
The paper investigates whether sparse autoencoder (SAE) features that recur across different language settings in Gemma 2 and Gemma 3 actually have consistent causal effects on translation performance. By reproducing Wu et al.’s discovery method and extending it to multilingual prompts, the authors find over 20 frequently activating features, yet causal validation reveals that almost all have negligible or inconsistent impacts. Only one feature—Gemma 2’s (L10, 5717) and Gemma 3’s (L20, 2456)—consistently improves COMET scores when amplified and worsens them when ablated across 23 language settings, indicating a language‑agnostic translation‑initiation direction.
By Giang Son Nguyen, Nhi Ngoc-Yen Nguyen, Wray Buntine, Dung D. Le
arXiv:2609.08322v1 Announce Type: cross
Abstract: Multilingual LLMs show stereotype-related behavior that varies across languages, but behavioral scores do not show where the relevant information is...
By Ariun-Erdene Tumurchuluun, Yusser Al Ghussin, Pinzhen Chen, Josef van Genabith, Koel Dutta Chowdhury
arXiv:2604. 03532v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show strong multilingual capabilities, yet reliably controlling the language of their outputs remains difficult.
By Sing Hieng Wong, Hassan Sajjad, A. B. Siddique
arXiv:2607. 19386v1 Announce Type: new Abstract: Cross-paper comparison of sparse autoencoder (SAE) interpretability often relies on autointerpretability scores.
By Sinie van der Ben, Neele Roch, Anna Hedstr\"om, Mennatallah El-Assady
Large language models show strong cross‑lingual abilities, but previous studies have examined them only at isolated training points. This work pretrains a 1.7B multilingual model on nine languages, recording checkpoints at fine granularity to track translation development. Using word‑level translation and a new dataset, the authors find that early learning is dominated by token‑level copying and surface similarities, while later stages involve more general translation mechanisms and refined copying.
By Felicia K\"orner, Maria Matveev, Florian Eichin, Gitta Kutyniok, Barbara Plank, Michael A. Hedderich
The paper investigates how multilingual large language models (LLMs) encode and express stereotypes across different languages. By applying linear probing, attribution patching, sparse autoencoders (SAEs), and feature ablation to Llama‑3.1‑8B, Qwen3‑8B, and Gemma‑2‑9B, the authors find that probe performance peaks much earlier than attribution, indicating a separation of 36‑53% of model depth. They observe that only a small fraction (6‑18%) of residual‑stream features exhibit language‑agnostic effects, and none are category‑agnostic, highlighting the need to measure decodability, output influence, and cross‑lingual ablation effects separately.
The paper evaluates four metrics—CKA, ANC, GMM dominance per token, and ILO—used to measure cross‑lingual representation sharing in multilingual language models. Across 21 models ranging from 125 M to 14 B parameters, the metrics disagree, and the authors attribute this to anisotropy, where representations cluster in a narrow embedding cone. Only ILO shows a strong, robust correlation with cross‑lingual transfer performance (Spearman’s ρ = 0.90) after controlling for model size, family, and task variation, leading the authors to recommend ILO as the primary metric alongside anisotropy diagnostics.
By Oskar Holmstr\"om, Marcel Bollmann, Marco Kuhlmann
arXiv:2606. 18383v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) are increasingly used to extract interpretable features from language models (LMs), yet a central question remains: when can an SAE-based explanation be treated as a faithful view of an underlying frozen LM We study this through a post-hoc generalization framework that certifies the LM via a sparse proxy, obtained by replacing a native hidden activation with its pretrained SAE reconstruction.
By Dibyanayan Bandyopadhyay, Asif Ekbal
arXiv:2608.21384v1 Announce Type: cross
Abstract: Modern multilingual tokenizers often fragment Ukrainian and other underrepresented Cyrillic-script languages more heavily than English, creating disp...
By Ivan Dobrovolskyi
The paper proposes the interlingua hypothesis, suggesting that large language models translate by encoding a source sentence into a latent, task‑agnostic feature space and then decoding a target sentence from that space. Three lines of evidence support this: (1) BLEU variance across language pairs is largely explained by language‑specific competences without pair‑specific interactions; (2) many model components influence both monolingual and translation tasks; and (3) fine‑tuning on monolingual data recovers most translation gains seen with aligned documents. These findings converge to support the hypothesis and point toward new ways to understand and improve LLM translation.
By Jacob Brinton, Jannik Brinkmann, Mark Crovella, Aaron Mueller