← Back to all news
arXiv Computation and Language By Shin Yamaguchi, Masaya Taniguchi, Tatsuki Kuribayashi, Taro Watanabe, Yoshinobu Kano

Once a Tree, Always a Tree? Cross-lingual Transfer of Hierarchical Generalization in Language Models

Read the original on arXiv Computation and Language →

The paper investigates whether language models can transfer hierarchical syntactic knowledge from a source language (L1) to a target language (L2) using a novel cross‑lingual poverty‑of‑stimulus (CPOS) experimental design. Results show that models can partially transfer L1 structural evidence to L2, producing a small but detectable hierarchical bias that varies across language pairs. The study further demonstrates that vocabulary overlap is crucial for successful transfer, indicating that cross‑lingual learning of hierarchical generalization is conditionally achievable.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Sep 11

Structural priors for data-efficient language learning

The paper explores structural transfer, where models are first trained on non-language data such as music, probabilistic grammars, and cellular automata to induce priors for natural language tasks. This pretraining acts as a weight initialization for multilingual language modeling and leads to lower next-token prediction loss and smaller weight shifts during subsequent language training. However, the improved loss does not consistently translate into better downstream linguistic performance, and the efficiency of non-language data is lower than that of additional language data.

By Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager, Lisa Beinborn
benchmarks
More like this →
arXiv Computation and Language
Oct 12

Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining

Large language models show strong cross‑lingual abilities, but previous studies have examined them only at isolated training points. This work pretrains a 1.7B multilingual model on nine languages, recording checkpoints at fine granularity to track translation development. Using word‑level translation and a new dataset, the authors find that early learning is dominated by token‑level copying and surface similarities, while later stages involve more general translation mechanisms and refined copying.

By Felicia K\"orner, Maria Matveev, Florian Eichin, Gitta Kutyniok, Barbara Plank, Michael A. Hedderich
llms
More like this →
arXiv AI
Aug 28

Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics

The paper investigates why large language models sometimes hallucinate when asked about facts in a language different from the one in which the facts were learned. By training small Transformer models on synthetic multilingual datasets, the authors show that the degree of correlation between facts and their learning language (informativeness) and the ease of language identification (extractability) determine whether models develop unified or separate representations across languages. Unified representations enable cross‑lingual fact transfer, while separate representations do not. The study proposes a unifying perspective on cross‑lingual transfer and suggests training methods to promote representational unification.

By Carter Blum, Katja Filippova, Ann Yuan, Asma Ghandeharioun, Julian Zimmert, Fred Zhang, Jessica Hoffmann, Tal Linzen, Martin Wattenberg, Lucas Dixon, Mor Geva
llms
More like this →