arXiv AI
UltraBERT: Beyond the Limit of BERT Pre-Training
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 18
Rescaling MLM-Head for Neural Sparse Retrieval
arXiv:2606. 18811v1 Announce Type: cross Abstract: Learned sparse retrieval (LSR) models such as SPLADE have traditionally used BERT-style masked language models as backbone encoders.
arXiv AI
Jul 2
Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps
arXiv:2607. 00004v1 Announce Type: cross Abstract: While advanced foundation models like ModernBERT significantly outperform older architectures in dense retrieval, they surprisingly lag behind the aging BERT-base baseline in learned sparse retrieval (LSR).
arXiv Computation and Language
Sep 2
Polish ModernBERT: The Long and Short of Polish Language Understanding
Polish ModernBERT is a new family of encoder‑only Transformers for Polish, offering Base and Large models with both 512‑token and 8K‑token context variants. The authors adapted the ModernBERT pretraining recipe through staged selection experiments and released a long‑context benchmark covering legal topic classification, ideological decision‑direction prediction, factual‑consistency assessment over literary plot summaries, and human‑rights violation assessment. Across 30 tasks, Polish ModernBERT outperforms existing Polish encoders, achieving 83.99 and 85.11 on the Base‑8K and Large‑8K models, respectively, and improving long‑context performance while using fewer parameters and lower memory and latency.
arXiv Computation and Language
Oct 8
Tiny-Scale Chinese BERT Pretraining: A Controlled Comparison of MLM, WWM, and MacBERT Strategies
arXiv:2610.08879v1 Announce Type: new Abstract: Pretraining strategies significantly impact the quality of language models, yet existing comparisons of Masked Language Modeling (MLM), Whole Word Mask...
arXiv Machine Learning
Jun 2
Test-Time Compute for Frozen Embedding Models through Agentic Program Search
arXiv:2605. 11374v5 Announce Type: replace Abstract: Test-time compute is widely believed to benefit only large reasoning models, leaving small models with nothing to gain.
arXiv AI
Aug 11
To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval
arXiv:2604. 00715v2 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) improves language model (LM) performance by providing relevant context at test time for knowledge-intensive situations.