arXiv Machine Learning August 4, 2026 By Kenny Shao

Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding

Read the original on arXiv Machine Learning →

arXiv:2608. 00837v1 Announce Type: cross Abstract: Byte Pair Encoding (BPE) is widely used for subword tokenization, but standard BPE exposes every learned merge token to the downstream model, including tokens that mainly serve as intermediate construction units and rarely appear in the final encoded corpus.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Aug 11

Length-MAX Tokenizer for Language Models

arXiv:2511. 20849v2 Announce Type: replace-cross Abstract: We introduce a new tokenizer for language models that minimizes the average tokens per character, thereby reducing the number of tokens needed to represent text during training and to generate text during inference.

By Dong Dong, Weijie Su

llms rag nlp efficiency

arXiv AI

Jul 17

In-Place Tokenizer Expansion for Pre-trained LLMs

arXiv:2607. 15232v1 Announce Type: cross Abstract: A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time.

By Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera, Simon S. Lee, Paul Pak, Aditya Tadimeti, Tim Seyde, Maxime Labonne, Alexander Amini, Mathias Lechner

llms rag nlp

arXiv AI

Jul 10

Less Is More: Reducing Token Counts Without Compromising Performance

arXiv:2506. 15138v2 Announce Type: replace-cross Abstract: Tokenization directly affects the inference efficiency of large language models, since fragmented tokenization increases sequence length and generation cost.

By Gyeongje Cho, Yeonkyoung So, Sangmin Lee, Jaejin Lee

llms nlp

arXiv Machine Learning

Jul 27

Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization

arXiv:2607. 22334v1 Announce Type: new Abstract: Open-weight language models from different families exhibit complementary capabilities, motivating their consolidation into a compact student through on-policy distillation (OPD).

By Hao Wang, Kun Yuan, Wenlin Zhong, Minglei Zhang, Han Xiao, Ming Sun, Honggang Qi

llms nlp efficiency benchmarks

arXiv Machine Learning

Jul 28

BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis

arXiv:2607. 23319v1 Announce Type: cross Abstract: Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages.

By Poornima Kumaresan, Pavithra Muruganantham, Lakshmi Rajendran, Santhosh Sivasubramani

llms nlp benchmarks

arXiv AI

Jul 1

CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield

arXiv:2606. 31796v1 Announce Type: cross Abstract: We study three complementary techniques for training compute-efficient language models.

By Dohyeon Kwon, Youngjin Park

llms

Related stories

Length-MAX Tokenizer for Language Models

In-Place Tokenizer Expansion for Pre-trained LLMs

Less Is More: Reducing Token Counts Without Compromising Performance

Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization

BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis

CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield