arXiv Computation and Language By Dries Rooryck, Alex Cai, Yonatan Belinkov, David Alvarez-Melis, Kiant\'e Brantley

Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence

Read the original on arXiv Computation and Language →

The paper investigates whether training small decoder-only transformers on code‑switched text can induce cross‑lingual alignment. Using two 100‑million‑word multilingual corpora—one a mix of English, Dutch, and Chinese BabyBabelLM data, and another generated by inserting word‑ and sentence‑level code‑switching via an LLM—the authors find that code‑switched training aligns representations of parallel text, especially across different scripts, and that this alignment persists when later training on monolingual documents. A curriculum that progresses from word‑level code‑switching to sentence‑level code‑switching and finally to monolingual data yields models that outperform baselines on the BabyLM evaluation suite, demonstrating that code‑switching curriculum learning is an effective data augmentation strategy for multilingual pretraining.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Computation and Language
Aug 28

Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

The study investigates whether monolingual language models, trained without joint multilingual objectives, develop cross-lingual alignment. By evaluating models such as Goldfish and independently built monolingual systems, the authors find that alignable representational geometry emerges across layers, strengthening with larger data, larger models, or closer linguistic proximity. A single Procrustes rotation on parallel sentences can map hidden states between models, and applying this rotation to a German model’s residuals swaps factual predictions to those of the donor English model, demonstrating functional transfer.

By Ej Zhou, Suchir Salhan, Catherine Arnett, Anna Korhonen
arXiv Computation and Language
Sep 2

Latent Mechanisms of Language Control in Multilingual Language Models

The paper investigates how multilingual large language models can unintentionally switch languages during generation. It compares three techniques—ValSel, FreqSel, and AnnSel—for pinpointing latent variables that control language choice in cross‑layer transcoders. Using new multilingual benchmarks and targeted interventions on Gemma‑2‑2B and Qwen3‑4B, the study finds all methods can steer output language, with FreqSel performing best and AnnSel providing interpretable selections via explicit annotations.

By Ryo Mitsuhashi, Sabri Boughorbel, Majd Hawasly
arXiv Computation and Language
Aug 28

Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations

The study compares an English-only and a bilingual decoder-only model, each 310 M parameters, trained on eight diverse languages while controlling for English exposure, compute, and document overlap. After aligning on shared English vocabulary, the authors find that token embeddings appear similar, but the deeper hidden states used for prediction diverge across models. This hidden‑state mismatch grows through middle transformer layers and persists despite controls, indicating that contextual processing differs between the models. "whyItMatters":"The findings show that embedding alignment can conceal significant internal representation differences, which is crucial for any downstream work that assumes aligned multilingual models are interchangeable."

By Anjishnu Mukherjee, Ziwei Zhu, Antonios Anastasopoulos
arXiv Computation and Language
Oct 22

How to Use Parallel Data for Multilingual Pretraining

The paper explores how to leverage parallel corpora for multilingual pretraining by concatenating each document with its translation in the same training sequence, a method called Parallel Pretraining (PPT). Compared to standard pretraining (SPT), PPT alone does not consistently improve language understanding, but it builds a stronger cross‑lingual foundation. A two‑stage approach—first PPT to establish cross‑lingual knowledge, then SPT to fine‑tune for standard language modeling—consistently outperforms standard pretraining in both high‑resource and low‑resource settings.

By Zhihong Huang, Yao Lu, Derek F. Wong, Hamid Alinejad-Rokny, Min Yang
arXiv Computation and Language
Aug 27

Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation

The paper investigates how to fairly compare language models across languages, noting that current evaluation methods vary widely and lack empirical validation. By training controlled monolingual models on parallel data and testing multilingual LLMs, the authors find that many normalized metrics suffer from biases due to tokenization, encoding, and orthographic differences. Instead, they recommend using sentence‑level negative log‑likelihood over semantically equivalent sequences for more reliable cross‑lingual comparisons.

By Xiulin Yang, Ethan Gotlieb Wilcox, Catherine Arnett