The paper investigates how multilingual large language models (LLMs) encode and express stereotypes across different languages. By applying linear probing, attribution patching, sparse autoencoders (SAEs), and feature ablation to Llama‑3.1‑8B, Qwen3‑8B, and Gemma‑2‑9B, the authors find that probe performance peaks much earlier than attribution, indicating a separation of 36‑53% of model depth. They observe that only a small fraction (6‑18%) of residual‑stream features exhibit language‑agnostic effects, and none are category‑agnostic, highlighting the need to measure decodability, output influence, and cross‑lingual ablation effects separately.
arXiv:2609.08322v1 Announce Type: cross Abstract: Multilingual LLMs show stereotype-related behavior that varies across languages, but behavioral scores do not show where the relevant information is...
The paper investigates whether sparse autoencoder (SAE) features that recur across different language settings in Gemma 2 and Gemma 3 actually have consistent causal effects on translation performance. By reproducing Wu et al.’s discovery method and extending it to multilingual prompts, the authors find over 20 frequently activating features, yet causal validation reveals that almost all have negligible or inconsistent impacts. Only one feature—Gemma 2’s (L10, 5717) and Gemma 3’s (L20, 2456)—consistently improves COMET scores when amplified and worsens them when ablated across 23 language settings, indicating a language‑agnostic translation‑initiation direction.
arXiv:2607. 20596v1 Announce Type: new Abstract: Sparse autoencoder (SAE) features are used to interpret and steer large language models, yet whether a feature's causal role is stable across SAE families remains untested.
The paper evaluates four metrics—CKA, ANC, GMM dominance per token, and ILO—used to measure cross‑lingual representation sharing in multilingual language models. Across 21 models ranging from 125 M to 14 B parameters, the metrics disagree, and the authors attribute this to anisotropy, where representations cluster in a narrow embedding cone. Only ILO shows a strong, robust correlation with cross‑lingual transfer performance (Spearman’s ρ = 0.90) after controlling for model size, family, and task variation, leading the authors to recommend ILO as the primary metric alongside anisotropy diagnostics.
The study probes Gemma‑2‑9B‑IT with Sparse Autoencoders across English, Hebrew, and Russian to examine how multilingual LLMs handle informal register. By using a dataset of polysemous terms that appear in literal and informal contexts, the authors isolate pragmatic register processing from lexical cues. They discover a small, robust cross‑linguistic core that forms an informal register subspace, which becomes clearer in deeper layers and can causally shift output formality across all tested languages, even transferring zero‑shot to six unseen languages.
arXiv:2606. 18383v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) are increasingly used to extract interpretable features from language models (LMs), yet a central question remains: when can an SAE-based explanation be treated as a faithful view of an underlying frozen LM We study this through a post-hoc generalization framework that certifies the LM via a sparse proxy, obtained by replacing a native hidden activation with its pretrained SAE reconstruction.
arXiv:2608. 12334v1 Announce Type: cross Abstract: Despite the impressive multilingual capabilities of Large Language Models, the latent dynamics dictating language selection remain poorly understood.
arXiv:2608. 02486v1 Announce Type: cross Abstract: Open-source LLMs reliably name Zeus, Jupiter, and Thor, but recover their counterparts in less-represented traditions like Finnish, Slavic, Egyptian, or Chinese mythology far less consistently.
The paper proposes the interlingua hypothesis, suggesting that large language models translate by encoding a source sentence into a latent, task‑agnostic feature space and then decoding a target sentence from that space. Three lines of evidence support this: (1) BLEU variance across language pairs is largely explained by language‑specific competences without pair‑specific interactions; (2) many model components influence both monolingual and translation tasks; and (3) fine‑tuning on monolingual data recovers most translation gains seen with aligned documents. These findings converge to support the hypothesis and point toward new ways to understand and improve LLM translation.
arXiv:2607. 04814v1 Announce Type: cross Abstract: Extending automatic speech recognition (ASR) to low-resource African languages is constrained by the prohibitive demands of data collection at scale.
The study investigates whether multilingual language models solve mathematical problems using shared or language‑specific internal representations. By replaying reasoning traces across six languages and applying Centered Kernel Alignment, the authors identify layers with cross‑language alignment and train two sparse autoencoders— a baseline and a new Geometry‑Invariant SAE (GI‑SAE) that adds an InfoNCE loss to encourage language‑agnostic activations. Functional interchangeability tests reveal that higher geometric similarity does not always translate to interchangeable features, and cross‑language sharing varies by model and depth, with GI‑SAE amplifying existing structure in some models but not others.
arXiv:2607. 05679v1 Announce Type: cross Abstract: Language models (LMs) exhibit problematic biases, such as stereotypes.