arXiv Computation and Language By Kevin Glocker, K\"atriin Kukk, Romina Oji, Marcel Bollmann, Marco Kuhlmann, Jenny Kunz

Outgrowing the Curse of Multilinguality: Scaling Strategies for Efficient Language Adaptation

Read the original on arXiv Computation and Language →

The paper explores how scaling pretrained language models can efficiently adapt them to new target languages, addressing the curse of multilinguality that hampers performance in medium- and low-resource settings. Through extensive ablations matched for data and FLOPs, the authors show that upscaling from English base models improves target-language performance by alleviating capacity constraints, especially for higher-resourced languages. While an initial upscaling step in English benefits low-resource languages, directly scaling up on the target language is more compute‑efficient for higher-resource scenarios, and for multilingual continued pre‑training, scaling up with target data yields the best balance of data efficiency and reduced English forgetting.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Computation and Language
Oct 22

Scaling Multilingual Pretraining of Pixel Language Models

Pixel language models treat text as rendered images, enabling strong cross‑lingual transfer. This study trains MPIXEL models varying languages, scripts, and capacity while keeping per‑language compute constant, and evaluates them on five tasks across 83 languages. Results show that scaling improves performance on added languages and scripts, mitigates the curse of multilinguality, and outperforms subword and character baselines on typologically distant low‑resource languages, though transfer to unseen scripts does not benefit further from scaling.

By Ilker Kesen, Jenny Kunz, Kasra Malihi, Desmond Elliott
arXiv Computation and Language
Aug 27

One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography

The study evaluates how different input representations—orthographic text, IPA transcription, and romanization—affect cross‑lingual transfer in autoregressive multilingual language models. Across three model sizes and eight languages grouped into typologically motivated pairs, romanized pretraining consistently outperforms native orthography and IPA, especially as model scale increases. Fine‑tuning a text‑pretrained model on romanized data can harm performance on languages already covered by the base model, suggesting romanization should be integrated at pretraining rather than applied later.

By Muge Zhang, Aaron Jencks, Krishna Badikela, Yulia Tsvetkov, Sachin Kumar
arXiv Computation and Language
Oct 12

Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining

Large language models show strong cross‑lingual abilities, but previous studies have examined them only at isolated training points. This work pretrains a 1.7B multilingual model on nine languages, recording checkpoints at fine granularity to track translation development. Using word‑level translation and a new dataset, the authors find that early learning is dominated by token‑level copying and surface similarities, while later stages involve more general translation mechanisms and refined copying.

By Felicia K\"orner, Maria Matveev, Florian Eichin, Gitta Kutyniok, Barbara Plank, Michael A. Hedderich
arXiv Computation and Language
Oct 22

How to Use Parallel Data for Multilingual Pretraining

The paper explores how to leverage parallel corpora for multilingual pretraining by concatenating each document with its translation in the same training sequence, a method called Parallel Pretraining (PPT). Compared to standard pretraining (SPT), PPT alone does not consistently improve language understanding, but it builds a stronger cross‑lingual foundation. A two‑stage approach—first PPT to establish cross‑lingual knowledge, then SPT to fine‑tune for standard language modeling—consistently outperforms standard pretraining in both high‑resource and low‑resource settings.

By Zhihong Huang, Yao Lu, Derek F. Wong, Hamid Alinejad-Rokny, Min Yang