arXiv:2609.37891v1 Announce Type: cross
Abstract: Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--...
By Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois, Pavel Chizhov, Carlos Rosas-Hinostroza, Neil Si Smail, Benjamin Burtin, Hanna Shcharbakova, Ivan Yamshchikov, Anastasia Stasenko
The paper presents practical training recipes for looped language models, showing that a compute‑efficient pipeline can reduce the training budget from 7.7 T tokens to 310 B tokens while maintaining strong reasoning performance. In controlled experiments, a 1.4 B LoopLM outperforms a parameter‑matched dense model on 12 benchmarks, achieving significant gains on GSM8K, MATH, and DROP, and approaches a 3.9 B dense model at matched inference compute. The authors also provide a minimal recipe to convert pretrained dense models into looped ones, demonstrating improvements on Qwen3‑1.7B‑Base across multiple data regimes.
By Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin, Inessa Fedorova, Dmitry Bocharov, Yuliana Shakhvalieva, Maria Tikhonova, Valerii Ternovskii
EDU-Jev is a family of single‑pass decision models (0.8B, 2B, and 4B parameters) designed for educational tasks that require selecting from a fixed set of options. The models output a probability distribution over predefined choices without generating text, using a pointer head to score each option. In twelve education benchmarks, EDU‑Jev 4B achieved 67.6% accuracy, outperforming five commercial decision APIs by 9.4 percentage points and the best open 4B model by 18.7 points, while the 2B version can process about 150 decisions per second on a single H100 GPU.
By Jiazheng Li, Yulan He
Manacá-1B is a 1.72‑billion‑parameter, open decoder‑only language model trained from scratch for Brazilian Portuguese, released with a fully containerized, reproducible training pipeline and complete logs. The authors evaluate it against nine open baselines on four Portuguese benchmarks, reporting standard errors and paired significance tests, and find that Manacá-1B outperforms smaller models on LAMBADA‑PT while remaining competitive on commonsense completion. They also uncover a tokenizer‑related evaluation pitfall that can drastically lower accuracy and provide a simple fix, releasing all code, logs, and corrected tokenizer for full reproducibility.
By Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fabio Andre Machado Porto
arXiv:2606. 26671v1 Announce Type: new Abstract: Post-training alignment determines the reasoning and human preference following capabilities of large language models, yet most existing works withhold detailed data construction, filtering rules and training recipes, which hinders community reproducibility and lightweight model optimization.
By Qiaobo Hao, Yangqian Wu, Shunyi Wang, Zhongjian Zhang, Ziqun Li, Yayin He, Muqing Li, Chen Zhong
arXiv:2608. 20281v1 Announce Type: cross Abstract: Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time.
By Qian Kou, Xiaofeng Shi, Xiaosong Qiu, Hua Zhou