arXiv Computation and Language
2d ago

Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

The study investigates whether monolingual language models, trained without joint multilingual objectives, develop cross-lingual alignment. By evaluating models such as Goldfish and independently built monolingual systems, the authors find that alignable representational geometry emerges across layers, strengthening with larger data, larger models, or closer linguistic proximity. A single Procrustes rotation on parallel sentences can map hidden states between models, and applying this rotation to a German model’s residuals swaps factual predictions to those of the donor English model, demonstrating functional transfer.

By Ej Zhou, Suchir Salhan, Catherine Arnett, Anna Korhonen