arXiv AI

Discovering Cross-Language Reasoning Invariance in LLMs with Geometry-Invariant Sparse Autoencoders

The study investigates whether multilingual language models solve mathematical problems using shared or language‑specific internal representations. By replaying reasoning traces across six languages and applying Centered Kernel Alignment, the authors identify layers with cross‑language alignment and train two sparse autoencoders— a baseline and a new Geometry‑Invariant SAE (GI‑SAE) that adds an InfoNCE loss to encourage language‑agnostic activations. Functional interchangeability tests reveal that higher geometric similarity does not always translate to interchangeable features, and cross‑language sharing varies by model and depth, with GI‑SAE amplifying existing structure in some models but not others.

arXiv Computation and Language

2d ago

Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations

The study compares an English-only and a bilingual decoder-only model, each 310 M parameters, trained on eight diverse languages while controlling for English exposure, compute, and document overlap. After aligning on shared English vocabulary, the authors find that token embeddings appear similar, but the deeper hidden states used for prediction diverge across models. This hidden‑state mismatch grows through middle transformer layers and persists despite controls, indicating that contextual processing differs between the models. "whyItMatters":"The findings show that embedding alignment can conceal significant internal representation differences, which is crucial for any downstream work that assumes aligned multilingual models are interchangeable."

By Anjishnu Mukherjee, Ziwei Zhu, Antonios Anastasopoulos

llms rag safety

arXiv Computation and Language

2d ago

Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

The study investigates whether monolingual language models, trained without joint multilingual objectives, develop cross-lingual alignment. By evaluating models such as Goldfish and independently built monolingual systems, the authors find that alignable representational geometry emerges across layers, strengthening with larger data, larger models, or closer linguistic proximity. A single Procrustes rotation on parallel sentences can map hidden states between models, and applying this rotation to a German model’s residuals swaps factual predictions to those of the donor English model, demonstrating functional transfer.

By Ej Zhou, Suchir Salhan, Catherine Arnett, Anna Korhonen

llms safety

arXiv Machine Learning

Aug 4

LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering

arXiv:2604. 03532v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show strong multilingual capabilities, yet reliably controlling the language of their outputs remains difficult.

By Sing Hieng Wong, Hassan Sajjad, A. B. Siddique

llms benchmarks

arXiv AI

2d ago

Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics

The paper investigates why large language models sometimes hallucinate when asked about facts in a language different from the one in which the facts were learned. By training small Transformer models on synthetic multilingual datasets, the authors show that the degree of correlation between facts and their learning language (informativeness) and the ease of language identification (extractability) determine whether models develop unified or separate representations across languages. Unified representations enable cross‑lingual fact transfer, while separate representations do not. The study proposes a unifying perspective on cross‑lingual transfer and suggests training methods to promote representational unification.

By Carter Blum, Katja Filippova, Ann Yuan, Asma Ghandeharioun, Julian Zimmert, Fred Zhang, Jessica Hoffmann, Tal Linzen, Martin Wattenberg, Lucas Dixon, Mor Geva

llms

arXiv AI

Jun 11

Cross-Layer Discrete Concept Discovery for Interpreting Language Models

arXiv:2506. 20040v3 Announce Type: replace-cross Abstract: Interpreting language models remains challenging due to the existence of residual stream, which linearly mixes and duplicates features across adjacent layers, causing single-layer analyses to miss this cross-layer structure.

By Ankur Garg, Xuemin Yu, Hassan Sajjad, Samira Ebrahimi Kahou

llms diffusion efficiency

arXiv AI

Aug 14

Gradual Code-Switching as Inference-Time Cross-Lingual Representational Alignment for LLMs

arXiv:2510. 05678v2 Announce Type: replace-cross Abstract: While large language models (LLMs) have achieved notable progress in multilingual settings, their performance remains uneven across languages as LLMs often rely on English-centric latent representations.

By Haneul Yoo, Jiho Jin, Kyunghyun Cho, Alice Oh

llms safety

arXiv Machine Learning

Aug 7

Cross-Architecture Steering Transfer in Language Models: A Systematic Empirical Study

arXiv:2608. 05164v1 Announce Type: cross Abstract: Independently trained large language models may develop shared internal representations of semantic concepts despite architectural differences -- but whether this geometric similarity has functional consequences for cross-model behavioural control remains untested.

By Ayushi Agarwal

llms fine-tuning safety

arXiv Machine Learning

Jun 18

From Sparse Features to Trustworthy Proxies: Certifying SAE-Based Interpretability

arXiv:2606. 18383v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) are increasingly used to extract interpretable features from language models (LMs), yet a central question remains: when can an SAE-based explanation be treated as a faithful view of an underlying frozen LM We study this through a post-hoc generalization framework that certifies the LM via a sparse proxy, obtained by replacing a native hidden activation with its pretrained SAE reconstruction.

By Dibyanayan Bandyopadhyay, Asif Ekbal

llms efficiency safety

arXiv Computation and Language

3d ago

Rethinking the Multilingual Reasoning Gap with Layer Swap

arXiv:2605.26735v2 Announce Type: replace Abstract: Recent reasoning Large Language Models produce a chain-of-thought (CoT) predominantly in English, even when prompted in non-English languages. Prio...

By Maxence Lasbordes, Am\'elie Chatelain, Djam\'e Seddah

llms fine-tuning

arXiv AI

Jul 28

Similarity All The Way Up: Multilingual Generalization in LLMs Relies on Language-Level Similarity Structures

arXiv:2607. 22699v1 Announce Type: new Abstract: As Large Language Models (LLMs) grow more capable across diverse tasks, their (in)ability to generalize remains difficult to quantify and poorly understood beyond limited domains.

By Supantho Rakshit, Adele Goldberg, Henry Conklin

llms benchmarks

arXiv AI

Jul 21

Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective

arXiv:2607. 18026v1 Announce Type: new Abstract: Can large language models with substantially different parameter spaces be merged by direct weighted averaging, without training or semantic alignment?

By Jiahe Fan, Yinghao Hou, Si Chen, Aiyuan Zhang, Hong Xie, Defu Lian

llms efficiency benchmarks safety

arXiv AI

2d ago

MIMO: Multilingual Information Retrieval via Monolingual Objectives

arXiv:2605.31171v2 Announce Type: replace-cross Abstract: Multilingual Information Retrieval (MLIR) reflects real-world search environments in which queries and relevant documents may appear in diffe...

By Youngjoon Jang, Seongtae Hong, Heuiseok Lim

rag efficiency benchmarks safety

Related stories