arXiv Computation and Language

Cross-Linguistic Effects in Bilingual Phoneme BabyLMs

arXiv Computation and Language
Oct 23

Once a Tree, Always a Tree? Cross-lingual Transfer of Hierarchical Generalization in Language Models

The paper investigates whether language models can transfer hierarchical syntactic knowledge from a source language (L1) to a target language (L2) using a novel cross‑lingual poverty‑of‑stimulus (CPOS) experimental design. Results show that models can partially transfer L1 structural evidence to L2, producing a small but detectable hierarchical bias that varies across language pairs. The study further demonstrates that vocabulary overlap is crucial for successful transfer, indicating that cross‑lingual learning of hierarchical generalization is conditionally achievable.

By Shin Yamaguchi, Masaya Taniguchi, Tatsuki Kuribayashi, Taro Watanabe, Yoshinobu Kano
arXiv AI
Oct 28

Specialization, Diversity, and Scaling in Child-Scale Language Models

The study trains small GPT‑2 models on child‑scale language data from dense home recordings (BabyView) and synthetic child‑directed dialogue (TinyDialogues), varying data amount, source, and diversity. Models trained on a single child’s environment predict that child’s language far better than others, a pattern largely driven by lexical and distributional factors as shown by a 5‑gram baseline. While training on multiple children improves predictions on pooled held‑out language, it does not yield a clear advantage on grammatical benchmarks, and naturalistic versus synthetic inputs produce distinct grammatical and lexical‑semantic patterns.

By Steven Y. Feng, Michael C. Frank
arXiv Computation and Language
Aug 28

Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations

The study compares an English-only and a bilingual decoder-only model, each 310 M parameters, trained on eight diverse languages while controlling for English exposure, compute, and document overlap. After aligning on shared English vocabulary, the authors find that token embeddings appear similar, but the deeper hidden states used for prediction diverge across models. This hidden‑state mismatch grows through middle transformer layers and persists despite controls, indicating that contextual processing differs between the models. "whyItMatters":"The findings show that embedding alignment can conceal significant internal representation differences, which is crucial for any downstream work that assumes aligned multilingual models are interchangeable."

By Anjishnu Mukherjee, Ziwei Zhu, Antonios Anastasopoulos
arXiv Computation and Language
Oct 12

Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining

Large language models show strong cross‑lingual abilities, but previous studies have examined them only at isolated training points. This work pretrains a 1.7B multilingual model on nine languages, recording checkpoints at fine granularity to track translation development. Using word‑level translation and a new dataset, the authors find that early learning is dominated by token‑level copying and surface similarities, while later stages involve more general translation mechanisms and refined copying.

By Felicia K\"orner, Maria Matveev, Florian Eichin, Gitta Kutyniok, Barbara Plank, Michael A. Hedderich
arXiv Computation and Language
Oct 23

Bilingual Language Models Capture Cross-Linguistic Influence in Second-Language Reading

The study examines whether bilingual language models can reflect the influence of a reader’s first language (L1) on second-language (L2) reading. By comparing bilingual models trained on an L1 that matches the readers’ backgrounds to monolingual English models and bilingual models trained on other L1s, the authors find that matching L1 improves predictions of L2 reading times, especially in late-stage eye-tracking measures and among intermediate-proficiency readers. The effect is modulated by the overlap between L1 and L2 scripts, indicating that bilingual models capture cross-linguistic influence in L2 reading behavior.

By Abderrahmane Issam, Irene E. Winther, Stefan L. Frank
arXiv Computation and Language
Sep 28

Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence

The paper investigates whether training small decoder-only transformers on code‑switched text can induce cross‑lingual alignment. Using two 100‑million‑word multilingual corpora—one a mix of English, Dutch, and Chinese BabyBabelLM data, and another generated by inserting word‑ and sentence‑level code‑switching via an LLM—the authors find that code‑switched training aligns representations of parallel text, especially across different scripts, and that this alignment persists when later training on monolingual documents. A curriculum that progresses from word‑level code‑switching to sentence‑level code‑switching and finally to monolingual data yields models that outperform baselines on the BabyLM evaluation suite, demonstrating that code‑switching curriculum learning is an effective data augmentation strategy for multilingual pretraining.

By Dries Rooryck, Alex Cai, Yonatan Belinkov, David Alvarez-Melis, Kiant\'e Brantley
arXiv Machine Learning
Jun 30

BabyHuBERT: Multilingual Self-Supervised Learning for Segmenting Speakers in Child-Centered Long-Form Recordings

arXiv:2509. 15001v3 Announce Type: replace-cross Abstract: Child-centered daylong recordings are essential for studying early language development, but existing speech models trained on clean adult data perform poorly due to acoustic and linguistic differences.

By Th\'eo Charlot, Tarek Kunze, Maxime Poli, Alejandrina Cristia, Emmanuel Dupoux, Marvin Lavechin
arXiv Machine Learning
Oct 7

A theory of platonic representations in language models

arXiv:2610.07168v1 Announce Type: new Abstract: Representations of translated sentences are similar in the inner layers of multilingual language models -- an observation connected to the platonic rep...

By Darshil Doshi, Wenjie Zhou, Corinna Elena Wegner, Daniel J. Korchinski, Santiago Acevedo, Matthieu Wyart
arXiv Machine Learning
Oct 9

Phonological Interference in Multilingual Speech Models

The paper investigates a failure mode called phonological interference in phoneme‑level multilingual speech models, where models incorrectly impose a single language’s phonology on mixed or unseen language input. Experiments show significant loss of language‑specific phonemes in code‑switched and unseen‑language scenarios, and the authors trace this effect to a low‑dimensional subspace of internal activations. They propose a windowed language estimation (WLE) repair that reduces interference by 34–69% while preserving monolingual performance.

By Moran Yanuka, Raja Giryes, Moris Alper