arXiv AI
Jul 7
Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
arXiv:2508. 04796v3 Announce Type: replace-cross Abstract: Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines.
Tokenisation over Bounded Alphabets is Hard
arXiv:2511. 15709v2 Announce Type: replace-cross Abstract: Recent works have shown that tokenisation is NP-complete.
Related stories
arXiv AI
Aug 11
Length-MAX Tokenizer for Language Models
arXiv:2511. 20849v2 Announce Type: replace-cross Abstract: We introduce a new tokenizer for language models that minimizes the average tokens per character, thereby reducing the number of tokens needed to represent text during training and to generate text during inference.
arXiv Machine Learning
22h ago
Large Language Models: A Mathematical Formulation
arXiv:2601. 22170v2 Announce Type: replace-cross Abstract: Large language models (LLMs) process and predict sequences containing text to answer questions, and address tasks including document summarization, providing recommendations, writing software and solving quantitative problems.
arXiv AI
Jul 10
Less Is More: Reducing Token Counts Without Compromising Performance
arXiv:2506. 15138v2 Announce Type: replace-cross Abstract: Tokenization directly affects the inference efficiency of large language models, since fragmented tokenization increases sequence length and generation cost.
arXiv Machine Learning
Jul 28
BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis
arXiv:2607. 23319v1 Announce Type: cross Abstract: Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages.
arXiv Machine Learning
Jun 2
Byte Pair Encoding for Efficient Time Series Forecasting
arXiv:2505. 14411v4 Announce Type: replace Abstract: Existing time series tokenization methods predominantly encode a constant number of samples into individual tokens.
arXiv AI
5d ago
Trie Automata for Constrained Decoding over Large Finite Sets
arXiv:2608. 12574v1 Announce Type: new Abstract: Large language models increasingly need to generate structured outputs that conform to predefined schemas, with one common constraint being selection from a finite set of valid strings.
arXiv Machine Learning
Jul 15
Language Identification with Succinct Machine-Independent Traces
arXiv:2607. 12443v1 Announce Type: cross Abstract: Motivated by the power of large language models, there has been renewed interest in the Gold-Angluin model of language identification in the limit, with an eye toward variants of the model that might overcome the negative results for its original formulation.
arXiv AI
Jun 2
You Can Learn Tokenization End-to-End with Reinforcement Learning
arXiv:2602. 13940v2 Announce Type: replace-cross Abstract: Tokenization is a hardcoded compression step which remains in the training pipeline of Large Language Models (LLMs), despite a general trend towards architectures becoming increasingly end-to-end.
arXiv Machine Learning
Jun 3
Names Don't Matter: Symbol-Invariant Transformer for Open-Vocabulary Learning
arXiv:2601. 23169v2 Announce Type: replace Abstract: Current neural architectures lack a principled way to handle interchangeable tokens, i.
arXiv Machine Learning
Jul 8
Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES
arXiv:2607. 05691v1 Announce Type: cross Abstract: Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny.
arXiv Machine Learning
Jul 27
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
arXiv:2607. 22334v1 Announce Type: new Abstract: Open-weight language models from different families exhibit complementary capabilities, motivating their consolidation into a compact student through on-policy distillation (OPD).