arXiv:2610.33583v1 Announce Type: new Abstract: Multilingual BabyLM systems usually fix language sampling proportions even though the weakest language can change during training. We present ARMOR-DP,...
The paper investigates whether training small decoder-only transformers on code‑switched text can induce cross‑lingual alignment. Using two 100‑million‑word multilingual corpora—one a mix of English, Dutch, and Chinese BabyBabelLM data, and another generated by inserting word‑ and sentence‑level code‑switching via an LLM—the authors find that code‑switched training aligns representations of parallel text, especially across different scripts, and that this alignment persists when later training on monolingual documents. A curriculum that progresses from word‑level code‑switching to sentence‑level code‑switching and finally to monolingual data yields models that outperform baselines on the BabyLM evaluation suite, demonstrating that code‑switching curriculum learning is an effective data augmentation strategy for multilingual pretraining.
arXiv:2410. 07809v2 Announce Type: replace-cross Abstract: Multilingual instruction tuning (MIT) is challenged by the curse of multilinguality, data scarcity, and high computational cost.
arXiv:2604.13286v2 Announce Type: replace Abstract: Despite the widespread multilingual deployment of large language models, post-training pipelines remain predominantly English-centric, contributing...
The paper explores how to leverage parallel corpora for multilingual pretraining by concatenating each document with its translation in the same training sequence, a method called Parallel Pretraining (PPT). Compared to standard pretraining (SPT), PPT alone does not consistently improve language understanding, but it builds a stronger cross‑lingual foundation. A two‑stage approach—first PPT to establish cross‑lingual knowledge, then SPT to fine‑tune for standard language modeling—consistently outperforms standard pretraining in both high‑resource and low‑resource settings.
arXiv:2610.20247v1 Announce Type: cross Abstract: Multilingual models share capacity across languages, so adding languages can hurt performance on existing ones. Mixture-of-experts models expand this...
The study evaluates how different input representations—orthographic text, IPA transcription, and romanization—affect cross‑lingual transfer in autoregressive multilingual language models. Across three model sizes and eight languages grouped into typologically motivated pairs, romanized pretraining consistently outperforms native orthography and IPA, especially as model scale increases. Fine‑tuning a text‑pretrained model on romanized data can harm performance on languages already covered by the base model, suggesting romanization should be integrated at pretraining rather than applied later.
arXiv:2607.00890v2 Announce Type: replace Abstract: Open web-scale pre-training corpora remain concentrated in English, limiting multilingual LLM development. We introduce MultiSynt/MT, an open synth...
Large language models show strong cross‑lingual abilities, but previous studies have examined them only at isolated training points. This work pretrains a 1.7B multilingual model on nine languages, recording checkpoints at fine granularity to track translation development. Using word‑level translation and a new dataset, the authors find that early learning is dominated by token‑level copying and surface similarities, while later stages involve more general translation mechanisms and refined copying.
The paper presents Qwenfant, a Qwen3-based model trained on concise parallel text for the 2026 BabyLM Challenge’s Multilingual track. By leveraging LLM-summarized and translated data, a larger model size, and a character-level vocabulary—both fine-tuned with LoRA—the authors achieve a top ranking in the competition.
The paper investigates why large language models (LLMs) fail to transfer knowledge across languages. By pretraining 360M- and 7B-parameter models, the authors show that poor cross‑lingual generalization arises during pretraining and persists despite standard fixes. Using a controlled bilingual setup with identical text but disjoint token spaces, they demonstrate that token disjointness alone causes knowledge compartmentalization, and that mapping languages into a shared token space via word‑wise translation markedly improves cross‑lingual performance, recovering up to 12.6% of native‑language learning efficiency.
arXiv:2601.16390v2 Announce Type: replace-cross Abstract: Large language models exhibit strong multilingual capabilities, yet significant performance gaps persist between dominant and non-dominant la...