arXiv Machine Learning August 28, 2026 By Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami Pun

Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility

Read the original on arXiv Machine Learning →

The paper shows that HuggingFace’s ByteLevel pre‑tokenizer, which treats a word as a sequence of Unicode letters, splits abugida scripts at every vowel sign, creating a training‑free lower bound on tokenizer fertility. Across 26 languages, all 17 abugidas exhibit increased token counts (up to 9×), while Latin, Cyrillic, Hangul, and Han remain unchanged. The authors demonstrate that correcting the character class reduces Nepali token counts, improves model performance, and that this issue is widespread in popular HuggingFace models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 28

BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis

arXiv:2607. 23319v1 Announce Type: cross Abstract: Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages.

By Poornima Kumaresan, Pavithra Muruganantham, Lakshmi Rajendran, Santhosh Sivasubramani

llms nlp benchmarks

arXiv AI

Jul 28

The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages

arXiv:2607. 24276v1 Announce Type: cross Abstract: Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words.

By Priyansh Srivastava

llms nlp benchmarks

Hugging Face Trending Papers

Jul 27

The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages

Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words. Because these tokenizers are trained predominantly on English-centric corpora, they introduce a systematic and often overlooked disadvantage for many non-English languages.

llms nlp benchmarks

arXiv AI

5d ago

Beyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI Systems

arXiv:2608.21384v1 Announce Type: cross Abstract: Modern multilingual tokenizers often fragment Ukrainian and other underrepresented Cyrillic-script languages more heavily than English, creating disp...

By Ivan Dobrovolskyi

rag nlp benchmarks

arXiv AI

Jul 10

Less Is More: Reducing Token Counts Without Compromising Performance

arXiv:2506. 15138v2 Announce Type: replace-cross Abstract: Tokenization directly affects the inference efficiency of large language models, since fragmented tokenization increases sequence length and generation cost.

By Gyeongje Cho, Yeonkyoung So, Sangmin Lee, Jaejin Lee

llms nlp

arXiv AI

Aug 11

Length-MAX Tokenizer for Language Models

arXiv:2511. 20849v2 Announce Type: replace-cross Abstract: We introduce a new tokenizer for language models that minimizes the average tokens per character, thereby reducing the number of tokens needed to represent text during training and to generate text during inference.

By Dong Dong, Weijie Su

llms rag nlp efficiency

One email a morning, machine-written

Related stories

BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis

The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages

The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages

Beyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI Systems

Less Is More: Reducing Token Counts Without Compromising Performance

Length-MAX Tokenizer for Language Models