arXiv Machine Learning August 5, 2026 By Raviraj Joshi, Utkarsh Vaidya, Sanjay Singh Chauhan, Niranjan Wartikar

Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension

Read the original on arXiv Machine Learning →

arXiv:2608. 03494v1 Announce Type: cross Abstract: Vocabulary extension is an efficient way to adapt pretrained large language models (LLMs) to new languages, but the initialization of newly added token embeddings can strongly affect continued pre-training (CPT) efficiency.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 17

In-Place Tokenizer Expansion for Pre-trained LLMs

arXiv:2607. 15232v1 Announce Type: cross Abstract: A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time.

By Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera, Simon S. Lee, Paul Pak, Aditya Tadimeti, Tim Seyde, Maxime Labonne, Alexander Amini, Mathias Lechner

llms rag nlp

arXiv AI

Aug 11

Length-MAX Tokenizer for Language Models

arXiv:2511. 20849v2 Announce Type: replace-cross Abstract: We introduce a new tokenizer for language models that minimizes the average tokens per character, thereby reducing the number of tokens needed to represent text during training and to generate text during inference.

By Dong Dong, Weijie Su

llms rag nlp efficiency

arXiv AI

Jul 14

Workload-Driven Optimization for On-Device Real-Time Subtitle Translation

arXiv:2607. 09957v1 Announce Type: cross Abstract: This report studies on-device English-to-Traditional-Chinese subtitle translation for Taiwan under short inputs, short outputs, batch-size-one inference, low latency, and privacy constraints.

By Tsz-To Wong

llms rag nlp fine-tuning efficiency benchmarks safety

arXiv AI

Jul 2

Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps

arXiv:2607. 00004v1 Announce Type: cross Abstract: While advanced foundation models like ModernBERT significantly outperform older architectures in dense retrieval, they surprisingly lag behind the aging BERT-base baseline in learned sparse retrieval (LSR).

By Zhichao Geng, Yang Yang

fine-tuning efficiency benchmarks

arXiv AI

Jun 6

Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning

arXiv:2606. 05173v1 Announce Type: cross Abstract: Masked language modelling (MLM) has been the dominant pre-training objective for text encoders since BERT, yet it encourages representations that are strongly anchored to surface-form token identity rather than deeper semantic structure.

By Aimen Boukhari

rag diffusion benchmarks

arXiv Machine Learning

Aug 4

LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering

arXiv:2604. 03532v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show strong multilingual capabilities, yet reliably controlling the language of their outputs remains difficult.

By Sing Hieng Wong, Hassan Sajjad, A. B. Siddique

llms benchmarks

Related stories

In-Place Tokenizer Expansion for Pre-trained LLMs

Length-MAX Tokenizer for Language Models

Workload-Driven Optimization for On-Device Real-Time Subtitle Translation

Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps

Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning

LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering