← Back to all news
arXiv AI

The Interlingua Hypothesis: LLMs Translate via a Latent Task-agnostic Feature Space

The paper proposes the interlingua hypothesis, suggesting that large language models translate by encoding a source sentence into a latent, task‑agnostic feature space and then decoding a target sentence from that space. Three lines of evidence support this: (1) BLEU variance across language pairs is largely explained by language‑specific competences without pair‑specific interactions; (2) many model components influence both monolingual and translation tasks; and (3) fine‑tuning on monolingual data recovers most translation gains seen with aligned documents. These findings converge to support the hypothesis and point toward new ways to understand and improve LLM translation.

Related stories

arXiv Computation and Language
Oct 12

Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining

Large language models show strong cross‑lingual abilities, but previous studies have examined them only at isolated training points. This work pretrains a 1.7B multilingual model on nine languages, recording checkpoints at fine granularity to track translation development. Using word‑level translation and a new dataset, the authors find that early learning is dominated by token‑level copying and surface similarities, while later stages involve more general translation mechanisms and refined copying.

By Felicia K\"orner, Maria Matveev, Florian Eichin, Gitta Kutyniok, Barbara Plank, Michael A. Hedderich
llms
More like this →
arXiv Computation and Language
Aug 28

Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations

The study compares an English-only and a bilingual decoder-only model, each 310 M parameters, trained on eight diverse languages while controlling for English exposure, compute, and document overlap. After aligning on shared English vocabulary, the authors find that token embeddings appear similar, but the deeper hidden states used for prediction diverge across models. This hidden‑state mismatch grows through middle transformer layers and persists despite controls, indicating that contextual processing differs between the models. "whyItMatters":"The findings show that embedding alignment can conceal significant internal representation differences, which is crucial for any downstream work that assumes aligned multilingual models are interchangeable."

By Anjishnu Mukherjee, Ziwei Zhu, Antonios Anastasopoulos
llmsragsafety
More like this →
arXiv AI
Oct 1

On the (In)effectiveness of AMR Augmentation for Large Language Models

The paper investigates whether adding Abstract Meaning Representation (AMR) data to large language models (LLMs) improves performance on downstream tasks. By reproducing recent studies and applying a consistent hyperparameter protocol, the authors find that text-only baselines match or surpass AMR-augmented models. A perplexity-based probe shows that AMR does not provide LLMs with additional relational knowledge, suggesting no clear benefit from AMR augmentation.

By Hoa Quynh Nhung Nguyen, Jacopo Staiano, Michael Sullivan
llmsnlp
More like this →
arXiv AI
Sep 7

Recurrence Is Not Enough: Causally Validating Multilingual SAE Translation Features in Gemma 2 and 3

The paper investigates whether sparse autoencoder (SAE) features that recur across different language settings in Gemma 2 and Gemma 3 actually have consistent causal effects on translation performance. By reproducing Wu et al.’s discovery method and extending it to multilingual prompts, the authors find over 20 frequently activating features, yet causal validation reveals that almost all have negligible or inconsistent impacts. Only one feature—Gemma 2’s (L10, 5717) and Gemma 3’s (L20, 2456)—consistently improves COMET scores when amplified and worsens them when ablated across 23 language settings, indicating a language‑agnostic translation‑initiation direction.

By Giang Son Nguyen, Nhi Ngoc-Yen Nguyen, Wray Buntine, Dung D. Le
More like this →
arXiv AI
Sep 2

Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs

The paper investigates whether different latent language probes—GMM-based representation probes and decoding-based probes—measure the same phenomenon in multilingual language models. Across various model families, training regimes, domains, tasks, checkpoints, and up to 27 languages, the authors find systematic disagreement: representation probes indicate earlier cross‑lingual mixing, while decoding probes reveal sharper, English‑biased language signals. These differences correlate with model multilinguality and training progression but remain relatively stable across domains, suggesting that current probes capture distinct aspects of multilingual processing rather than a single internal lingua franca.

By Deniz Bayazit, Badr AlKhamissi, Antoine Bosselut
llms
More like this →