arXiv AI
Cross-Lingual Activation Steering for Multilingual Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 4
One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging
The paper investigates weight‑space merging of independently fine‑tuned multilingual machine translation models. Experiments show that merging is more successful when models share a target language, yet it still cannot match the peak performance of language‑specific checkpoints. When target languages differ, performance drops sharply, and analysis reveals that overlapping neuron activation and incompatible upper‑layer geometries cause these failures.
arXiv Computation and Language
Aug 28
Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations
The study compares an English-only and a bilingual decoder-only model, each 310 M parameters, trained on eight diverse languages while controlling for English exposure, compute, and document overlap. After aligning on shared English vocabulary, the authors find that token embeddings appear similar, but the deeper hidden states used for prediction diverge across models. This hidden‑state mismatch grows through middle transformer layers and persists despite controls, indicating that contextual processing differs between the models. "whyItMatters":"The findings show that embedding alignment can conceal significant internal representation differences, which is crucial for any downstream work that assumes aligned multilingual models are interchangeable."
arXiv AI
Sep 18
Why Pretraining Fails to Share Cross-Lingual Knowledge
The paper investigates why large language models (LLMs) fail to transfer knowledge across languages. By pretraining 360M- and 7B-parameter models, the authors show that poor cross‑lingual generalization arises during pretraining and persists despite standard fixes. Using a controlled bilingual setup with identical text but disjoint token spaces, they demonstrate that token disjointness alone causes knowledge compartmentalization, and that mapping languages into a shared token space via word‑wise translation markedly improves cross‑lingual performance, recovering up to 12.6% of native‑language learning efficiency.
arXiv Computation and Language
Sep 11
Distribution-aware Language Neuron Identification in Multilingual Large Language Models
The paper introduces a new method for identifying language-specific neurons in multilingual large language models (mLLMs). Unlike previous entropy-based approaches that only consider positive activations, the proposed Distribution-aware Language Neuron selection uses pairwise overlap coefficients of full activation distributions, including negative values, to cluster languages. Experiments on two mLLMs and two held-out corpora show that this method isolates language-specific causal effects more effectively, achieving up to 4.9× higher on-target language damage per neuron while maintaining off-target language performance.
arXiv Computation and Language
Aug 28
Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
The study investigates whether monolingual language models, trained without joint multilingual objectives, develop cross-lingual alignment. By evaluating models such as Goldfish and independently built monolingual systems, the authors find that alignable representational geometry emerges across layers, strengthening with larger data, larger models, or closer linguistic proximity. A single Procrustes rotation on parallel sentences can map hidden states between models, and applying this rotation to a German model’s residuals swaps factual predictions to those of the donor English model, demonstrating functional transfer.
arXiv Computation and Language
Aug 28
Cross-lingual Representation Learning via Centroid Intervention Fusion
The paper introduces Centroid Intervention Fusion (CIF), a framework that merges multiple multilingual intervention projections into a single language-shared operator for inference-time modification of large language models. CIF improves cross-lingual transfer without updating model parameters and achieves up to +3.378 percentage points better performance than prior pairwise intervention baselines across several benchmarks, including low-resource languages. The authors provide code at https://github.com/VRCMF/CIF.git.