arXiv AI
Jun 18
Rescaling MLM-Head for Neural Sparse Retrieval
arXiv:2606. 18811v1 Announce Type: cross Abstract: Learned sparse retrieval (LSR) models such as SPLADE have traditionally used BERT-style masked language models as backbone encoders.
UltraBERT: Beyond the Limit of BERT Pre-Training
Related stories
arXiv AI
Jul 2
Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps
arXiv:2607. 00004v1 Announce Type: cross Abstract: While advanced foundation models like ModernBERT significantly outperform older architectures in dense retrieval, they surprisingly lag behind the aging BERT-base baseline in learned sparse retrieval (LSR).
arXiv Computation and Language
Sep 2
Polish ModernBERT: The Long and Short of Polish Language Understanding
Polish ModernBERT is a new family of encoder‑only Transformers for Polish, offering Base and Large models with both 512‑token and 8K‑token context variants. The authors adapted the ModernBERT pretraining recipe through staged selection experiments and released a long‑context benchmark covering legal topic classification, ideological decision‑direction prediction, factual‑consistency assessment over literary plot summaries, and human‑rights violation assessment. Across 30 tasks, Polish ModernBERT outperforms existing Polish encoders, achieving 83.99 and 85.11 on the Base‑8K and Large‑8K models, respectively, and improving long‑context performance while using fewer parameters and lower memory and latency.
arXiv Computation and Language
Oct 8
Tiny-Scale Chinese BERT Pretraining: A Controlled Comparison of MLM, WWM, and MacBERT Strategies
arXiv:2610.08879v1 Announce Type: new Abstract: Pretraining strategies significantly impact the quality of language models, yet existing comparisons of Masked Language Modeling (MLM), Whole Word Mask...
arXiv Machine Learning
Jun 2
Test-Time Compute for Frozen Embedding Models through Agentic Program Search
arXiv:2605. 11374v5 Announce Type: replace Abstract: Test-time compute is widely believed to benefit only large reasoning models, leaving small models with nothing to gain.
arXiv AI
Aug 11
To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval
arXiv:2604. 00715v2 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) improves language model (LM) performance by providing relevant context at test time for knowledge-intensive situations.
arXiv Computation and Language
Oct 23
ULME: Predictive Coupling for Mutually Reinforcing Embedding and Language Modeling
ULME is a post‑training framework that unifies text embedding and language modeling within a single causal decoder‑only LLM by inserting learnable slots for semantic and reasoning embeddings. It trains three objectives—contrastive loss for semantic alignment, language‑modeling loss for answer generation, and a stop‑gradient predictive loss linking reasoning to the answer—so that generation actively shapes the embedding. Across Qwen2.5 and Qwen3 models (0.5B–8B), ULME improves MTEB scores by 3.3–4.1 points, tightens BRIGHT and AdvBench‑IR, and matches generation‑only fine‑tunes on standard accuracy benchmarks.
arXiv AI
Jul 31
ARC-Encoder: learning compressed text representations for large language models
arXiv:2510. 20535v2 Announce Type: replace-cross Abstract: Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs.
arXiv AI
Jul 17
In-Place Tokenizer Expansion for Pre-trained LLMs
arXiv:2607. 15232v1 Announce Type: cross Abstract: A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time.
arXiv Computation and Language
Oct 28
Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters
The paper presents a series of Icelandic language encoders built with the ModernBERT architecture, ranging from 12 million to 1 billion parameters, trained on 9 billion tokens of open data. It reexamines IceBERT’s findings, showing that a cleaned web crawl suffices for strong performance and that starting from a multilingual model can aid certain tasks. The 1 billion‑parameter model, trained from scratch, outperforms IceBERT on all eleven benchmark tasks, and when warm‑started from the English Ettin encoder it achieves the best scores across all evaluated tasks.
arXiv AI
Aug 7
Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers
arXiv:2608. 06111v1 Announce Type: cross Abstract: Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to \textit{syntactic structure}.
Hugging Face Trending Papers
Oct 22
ULME: Predictive Coupling for Mutually Reinforcing Embedding and Language Modeling
ULME introduces a post‑training framework that unifies text embedding and generation within a single causal decoder‑only LLM by inserting learnable slots for semantic and reasoning embeddings. It trains three objectives—contrastive, language‑modeling, and stop‑gradient predictive—so that generation actively shapes the embedding space. Across Qwen2.5 and Qwen3 models (0.5B–8B), ULME improves MTEB scores by 3.3–4.1 points, tightens BRIGHT and AdvBench‑IR, and matches generation‑only fine‑tunes on standard accuracy benchmarks.