MoganBert-TR is a 149‑million‑parameter Turkish encoder foundation model trained from scratch on a language‑specific corpus using a two‑stage CLM‑to‑MLM curriculum. The model, along with its embedding variant MoganBert‑Embed, achieves state‑of‑the‑art results on Turkish benchmarks such as TrGLUE and TabiBench, outperforming existing Turkish BERT models. Its tokenizer, comprising 50,048 tokens, also surpasses other Turkish tokenizers in compression and fertility metrics.
By Furkan Yilmaz, Habibe Aleyna Tasdemir, Muhammed Faruk Gozay
MoganColBERT-TR is a Turkish multi‑vector retrieval model that projects token embeddings from 768 to 128 dimensions and uses MaxSim late interaction for scoring. It builds on the previously trained MoganBERT‑TR encoder, adapting it to the ColBERT objective with a single‑epoch distillation phase using cross‑encoder teacher scores. Evaluated on five Turkish BEIR datasets in a zero‑shot setting, it achieves an overall score of 37.36, outperforming larger models on most datasets.
By Furkan Yilmaz, Habibe Aleyna Tasdemir, Muhammed Faruk Gozay
arXiv:2606. 28999v1 Announce Type: cross Abstract: Encoders have become the state of the art for multiple NLP tasks, especially those requiring deep contextual understanding.
By Renn\^e Ruan Alves Oliveira, Gustavo Cordeiro Galv\~ao Van Erven, Lu\'is Paulo Faina Garcia
arXiv:2606. 13647v1 Announce Type: cross Abstract: We introduce SkMTEB, the first comprehensive MTEB-style text embedding benchmark for Slovak, a low-resource West Slavic language, comprising 31 datasets across 7 task types -- nearly 4$\times$ the depth of existing multilingual benchmark coverage for Slovak.
By Marek \v{S}uppa, Andrej Ridzik, Daniel Hl\'adek, Nat\'alia K\v{n}a\v{z}ekov\'a, Vikt\'oria Ondrejov\'a
Polish ModernBERT is a new family of encoder‑only Transformers for Polish, offering Base and Large models with both 512‑token and 8K‑token context variants. The authors adapted the ModernBERT pretraining recipe through staged selection experiments and released a long‑context benchmark covering legal topic classification, ideological decision‑direction prediction, factual‑consistency assessment over literary plot summaries, and human‑rights violation assessment. Across 30 tasks, Polish ModernBERT outperforms existing Polish encoders, achieving 83.99 and 85.11 on the Base‑8K and Large‑8K models, respectively, and improving long‑context performance while using fewer parameters and lower memory and latency.
By Micha{\l} Pere{\l}kiewicz, S{\l}awomir Dadas, Rafa{\l} Po\'swiata, Ma{\l}gorzata Gr\k{e}bowiec
arXiv:2503. 05500v3 Announce Type: replace-cross Abstract: General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models.
By Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves, Andr\'e Martins, Ayoub Hammal, Caio Corro, C\'eline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, Jo\~ao Alves, Kevin El Haddad, Manuel Faysse, Maxime Peyrard, Nuno M. Guerreiro, Patrick Fernandes, Ricardo Rei, Pierre Colombo
State-of-the-art retrieval models increasingly rely on closed training data, creating a reproducibility gap. We present an open end-to-end recipe for training retrieval models and study how English supervision transfers to multilingual retrieval through translate-train.
arXiv:2012. 02110v2 Announce Type: replace-cross Abstract: Pre-trained language models have significantly advanced natural language processing (NLP), especially with the introduction of BERT and its optimized version, RoBERTa.
By Raphael Scheible, Johann Frei, Fabian Thomczyk, Henry He, Patric Tippmann, Jochen Knaus, Victor Jaravine, Frank Kramer, Martin Boeker
arXiv:2606. 31718v1 Announce Type: cross Abstract: Relation extraction (RE) for low-resource languages is typically constrained by the lack of annotated corpora.
By Dragos-Mitrut Vasile, Elena-Simona Apostol, Stefan-Adrian Toma, Adrian Paschke, Ciprian-Octavian Truica
arXiv:2606. 20089v1 Announce Type: cross Abstract: Persian pretrained language models (PLMs) are still limited by the scarcity of large-scale, high-quality pretraining corpora and by insufficient evaluation beyond standard classification and NER tasks.
By Arash Ghafouri, Mahdi Firouzmandi, Hossein Saberi, Mohammad Reza Hasani Ahangar
arXiv:2606. 18717v1 Announce Type: cross Abstract: Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text.
By Tolga \c{S}akar
arXiv:2510. 20535v2 Announce Type: replace-cross Abstract: Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs.
By Hippolyte Pilchen, Edouard Grave, Patrick P\'erez