The Flow has not summarised this story yet — read it at arXiv Computation and Language.
arXiv:2610.33583v1 Announce Type: new Abstract: Multilingual BabyLM systems usually fix language sampling proportions even though the weakest language can change during training. We present ARMOR-DP,...
The paper investigates whether training small decoder-only transformers on code‑switched text can induce cross‑lingual alignment. Using two 100‑million‑word multilingual corpora—one a mix of English, Dutch, and Chinese BabyBabelLM data, and another generated by inserting word‑ and sentence‑level code‑switching via an LLM—the authors find that code‑switched training aligns representations of parallel text, especially across different scripts, and that this alignment persists when later training on monolingual documents. A curriculum that progresses from word‑level code‑switching to sentence‑level code‑switching and finally to monolingual data yields models that outperform baselines on the BabyLM evaluation suite, demonstrating that code‑switching curriculum learning is an effective data augmentation strategy for multilingual pretraining.
arXiv:2410. 07809v2 Announce Type: replace-cross Abstract: Multilingual instruction tuning (MIT) is challenged by the curse of multilinguality, data scarcity, and high computational cost.
arXiv:2604.13286v2 Announce Type: replace Abstract: Despite the widespread multilingual deployment of large language models, post-training pipelines remain predominantly English-centric, contributing...
The paper explores how to leverage parallel corpora for multilingual pretraining by concatenating each document with its translation in the same training sequence, a method called Parallel Pretraining (PPT). Compared to standard pretraining (SPT), PPT alone does not consistently improve language understanding, but it builds a stronger cross‑lingual foundation. A two‑stage approach—first PPT to establish cross‑lingual knowledge, then SPT to fine‑tune for standard language modeling—consistently outperforms standard pretraining in both high‑resource and low‑resource settings.
arXiv:2610.20247v1 Announce Type: cross Abstract: Multilingual models share capacity across languages, so adding languages can hurt performance on existing ones. Mixture-of-experts models expand this...