← Back to all news
arXiv AI By Giang Son Nguyen, Nhi Ngoc-Yen Nguyen, Wray Buntine, Dung D. Le

Recurrence Is Not Enough: Causally Validating Multilingual SAE Translation Features in Gemma 2 and 3

Read the original on arXiv AI →

The paper investigates whether sparse autoencoder (SAE) features that recur across different language settings in Gemma 2 and Gemma 3 actually have consistent causal effects on translation performance. By reproducing Wu et al.’s discovery method and extending it to multilingual prompts, the authors find over 20 frequently activating features, yet causal validation reveals that almost all have negligible or inconsistent impacts. Only one feature—Gemma 2’s (L10, 5717) and Gemma 3’s (L20, 2456)—consistently improves COMET scores when amplified and worsens them when ablated across 23 language settings, indicating a language‑agnostic translation‑initiation direction.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Oct 12

Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining

Large language models show strong cross‑lingual abilities, but previous studies have examined them only at isolated training points. This work pretrains a 1.7B multilingual model on nine languages, recording checkpoints at fine granularity to track translation development. Using word‑level translation and a new dataset, the authors find that early learning is dominated by token‑level copying and surface similarities, while later stages involve more general translation mechanisms and refined copying.

By Felicia K\"orner, Maria Matveev, Florian Eichin, Gitta Kutyniok, Barbara Plank, Michael A. Hedderich
llms
More like this →
arXiv Computation and Language
Sep 25

LLM Forensics: Where Do Backdoors Hide? Localizing and Controlling Trigger Mechanisms with Sparse Autoencoders

The paper investigates how trigger-based backdoors function in large language models by using sparse autoencoders (SAEs) to identify feature directions across layers and transformer components. In a controlled experiment, 1B and 8B models were made to continue English prompts in French or German when presented with fixed trigger sequences. The study finds that different SAE feature directions correspond to trigger detection, residual-stream propagation, and language tracking, with residual-stream features being most effective for controlling the backdoor behavior.

By Wissam Antoun, Francis Kulumba, Th\'eo Lasnier, Beno\^it Sagot, Djam\'e Seddah
llms
More like this →
arXiv Computation and Language
Sep 2

Latent Mechanisms of Language Control in Multilingual Language Models

The paper investigates how multilingual large language models can unintentionally switch languages during generation. It compares three techniques—ValSel, FreqSel, and AnnSel—for pinpointing latent variables that control language choice in cross‑layer transcoders. Using new multilingual benchmarks and targeted interventions on Gemma‑2‑2B and Qwen3‑4B, the study finds all methods can steer output language, with FreqSel performing best and AnnSel providing interpretable selections via explicit annotations.

By Ryo Mitsuhashi, Sabri Boughorbel, Majd Hawasly
llmsbenchmarks
More like this →