MoganBert-TR is a 149‑million‑parameter Turkish encoder foundation model trained from scratch on a language‑specific corpus using a two‑stage CLM‑to‑MLM curriculum. The model, along with its embedding variant MoganBert‑Embed, achieves state‑of‑the‑art results on Turkish benchmarks such as TrGLUE and TabiBench, outperforming existing Turkish BERT models. Its tokenizer, comprising 50,048 tokens, also surpasses other Turkish tokenizers in compression and fertility metrics.
By Furkan Yilmaz, Habibe Aleyna Tasdemir, Muhammed Faruk Gozay
arXiv:2608. 20210v1 Announce Type: cross Abstract: Small language models are usually built like large ones and then squeezed onto a CPU afterwards.
By Christos Koutsiaris
arXiv:2607.09885v3 Announce Type: replace
Abstract: We present Index-1.9B, a series of open small language models developed at Bilibili. The series comprises four models: Index-1.9B-Base, a foundatio...
By Tianjiao Li, Lusheng Zhang, Shien He, Xiaojing Liu, Tianxing Yan, Mengran Yu, Ziang Cui, Kai Zhao, Xipeng Wang, Yang Liu, Yuxin Li
arXiv:2607. 19257v1 Announce Type: cross Abstract: Practitioners make three prompt-design decisions with almost no controlled evidence behind them: how to format instructions and context (markdown, plain text, prose, or tabular), how many simultaneous instructions a system prompt can carry before compliance degrades, and how much context a model can hold before recall and honesty degrade.
By Netanel Eliav
arXiv:2606. 18717v1 Announce Type: cross Abstract: Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text.
By Tolga \c{S}akar
Small language models are usually built like large ones and then squeezed onto a CPU afterwards. We did the opposite: we fixed the target first, one user, one token at a time, 4-bit weights, ordinary CPU, and chose the architecture to suit it.
arXiv:2607. 09424v1 Announce Type: cross Abstract: We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English.
By The Soofi-Team, :, Benedikt Droste, David Fitzek, Ruben H\"arle, Lukas Helff, Maximilian Idahl, Alex Jude, Abbas Goher Khan, Maurice Kraus, Timm Ruland, Richard Rutmann, Sebastian Sztwiertnia, Markus Frey, Daniil Gurgurov, Jan Pfister, Tom R\"ohr, Sebastian von Rohrscheidt, J\"org Bienert, Nicolas Flores-Herr, Simon Gottschalk, Andreas Hotho, Kristian Kersting, Joachim K\"ohler, Alexander L\"oser, Wolfgang Nejdl, Simon Ostermann, Jan Plogsties, Patrick Putzky, Mehdi Ali, Michael Fromm, Max L\"ubbering
The paper investigates whether small models distilled from larger ones behave similarly when using byte versus token tokenization. It introduces two methods—Marginalize‑It (approximate) and End‑Of‑Token (exact)—to convert token logits to byte logits, and conducts a large‑scale study on decoder‑only dense transformers ranging from 1 billion to 1 trillion bytes of data. Results show that while token‑based models excel early, byte‑based models eventually surpass them with more compute, achieving higher performance ceilings, greater data efficiency, and lower logit storage costs.
By Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz, Margaret Li, Mike Lewis, Luke Zettlemoyer, Srinivasan Iyer
The paper shows that HuggingFace’s ByteLevel pre‑tokenizer, which treats a word as a sequence of Unicode letters, splits abugida scripts at every vowel sign, creating a training‑free lower bound on tokenizer fertility. Across 26 languages, all 17 abugidas exhibit increased token counts (up to 9×), while Latin, Cyrillic, Hangul, and Han remain unchanged. The authors demonstrate that correcting the character class reduces Nepali token counts, improves model performance, and that this issue is widespread in popular HuggingFace models.
By Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami Pun
arXiv:2606. 24083v1 Announce Type: cross Abstract: "Talk short.
By Morayo Danielle Adeyemi, Ryan A. Rossi, Franck Dernoncourt
arXiv:2609.25008v1 Announce Type: new
Abstract: I pretrained a language model end-to-end in Rust - alone, with no team, no PyTorch, and no Python in the training path - for $164 in rented GPU time. I...
By Arif Adito
The paper introduces the first open Armenian large language model, arm‑gemma‑e4b, trained on two newly released datasets: ArmWeb, a 4.37 million‑document news corpus, and ArmSTEM, a 373 k English‑Armenian math and science problem set with verified step‑by‑step solutions. Continued pretraining of Gemma‑4‑E4B on these datasets outperforms all existing open Armenian models and demonstrates that adding a small portion of verified translated STEM data can restore knowledge lost during news‑only pretraining. The authors also reveal significant overlap between major public Armenian corpora and web‑derived evaluation panels, and they provide all data, models, and code openly.
By Erik Arakelyan, Khatun Avetisyan, Meri Davtyan, Heghine Grigoryan, Nane Khachatryan, Hayk Shahsuvaryan, Henrik Sergoyan, Vahan Martirosyan