arXiv:2511. 20849v2 Announce Type: replace-cross Abstract: We introduce a new tokenizer for language models that minimizes the average tokens per character, thereby reducing the number of tokens needed to represent text during training and to generate text during inference.
By Dong Dong, Weijie Su
arXiv:2607. 15232v1 Announce Type: cross Abstract: A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time.
By Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera, Simon S. Lee, Paul Pak, Aditya Tadimeti, Tim Seyde, Maxime Labonne, Alexander Amini, Mathias Lechner
arXiv:2607. 22334v1 Announce Type: new Abstract: Open-weight language models from different families exhibit complementary capabilities, motivating their consolidation into a compact student through on-policy distillation (OPD).
By Hao Wang, Kun Yuan, Wenlin Zhong, Minglei Zhang, Han Xiao, Ming Sun, Honggang Qi
The study investigates how the choice of tokenizer influences multilingual language models across 54 tokenizers and 123 models. It finds that tokenizer impact is greater for languages with less training data, and that excluding a language from tokenizer training consistently worsens its performance. While reallocating tokenizer training data to lower-resource languages can help, it does not guarantee improvement, and the best tokenizer properties vary by language, enabling predictive screening of tokenizer candidates.
By Clara Meister, G\"ul Sena Alt{\i}nta\c{s}, Antoine Bosselut
TurboMamba is a new language‑model architecture that combines a global Mamba‑3 model with a small Transformer to achieve faster inference, faster training, and better token prediction. It introduces three lightweight enhancements—direct access to the previous block’s last token, a small correction to next‑token scores, and a learned mixture with recent‑token frequencies—to improve local predictions. In experiments on FineWeb‑Edu, TurboMamba outperforms Transformer, Mamba‑3, Jamba, and Zamba, achieving the lowest mean test prediction loss and significantly higher training throughput and batch latency speedups.
By Hanxin Zhang
arXiv:2508.11857v3 Announce Type: replace-cross
Abstract: Tokenization remains a persistent bottleneck in language modeling, especially when vocabulary learning is limited by whitespace boundaries. W...
By Andrei-Valentin T\u{a}nase, Elena Pelican