← Back to all news
arXiv Computation and Language By Lukas Edman, Alexander Fraser

Qwenfant: Train Big Character-level BabyLMs on Concise Parallel Text

Read the original on arXiv Computation and Language →

The paper presents Qwenfant, a Qwen3-based model trained on concise parallel text for the 2026 BabyLM Challenge’s Multilingual track. By leveraging LLM-summarized and translated data, a larger model size, and a character-level vocabulary—both fine-tuned with LoRA—the authors achieve a top ranking in the competition.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Sep 28

Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence

The paper investigates whether training small decoder-only transformers on code‑switched text can induce cross‑lingual alignment. Using two 100‑million‑word multilingual corpora—one a mix of English, Dutch, and Chinese BabyBabelLM data, and another generated by inserting word‑ and sentence‑level code‑switching via an LLM—the authors find that code‑switched training aligns representations of parallel text, especially across different scripts, and that this alignment persists when later training on monolingual documents. A curriculum that progresses from word‑level code‑switching to sentence‑level code‑switching and finally to monolingual data yields models that outperform baselines on the BabyLM evaluation suite, demonstrating that code‑switching curriculum learning is an effective data augmentation strategy for multilingual pretraining.

By Dries Rooryck, Alex Cai, Yonatan Belinkov, David Alvarez-Melis, Kiant\'e Brantley
llmssafety
More like this →
arXiv AI
Jun 2

EuroBERT: Scaling Multilingual Encoders for European Languages

arXiv:2503. 05500v3 Announce Type: replace-cross Abstract: General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models.

By Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves, Andr\'e Martins, Ayoub Hammal, Caio Corro, C\'eline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, Jo\~ao Alves, Kevin El Haddad, Manuel Faysse, Maxime Peyrard, Nuno M. Guerreiro, Patrick Fernandes, Ricardo Rei, Pierre Colombo
More like this →