arXiv AI
SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 11
Length-MAX Tokenizer for Language Models
arXiv:2511. 20849v2 Announce Type: replace-cross Abstract: We introduce a new tokenizer for language models that minimizes the average tokens per character, thereby reducing the number of tokens needed to represent text during training and to generate text during inference.
arXiv Computation and Language
1d ago
Pearl-Chat: Engineering a native Luganda language model from first principles in JAX and Flax NNX
arXiv:2610.22529v1 Announce Type: new Abstract: Autoregressive foundation language models predominantly target high-resource Indo-European languages, imposing computational and morphological penaltie...
arXiv Machine Learning
Sep 10
Dynamic-Programming-Guided Hierarchical BPE and Empirical Analysis of Vocabulary Pruning
arXiv:2609.06898v1 Announce Type: cross Abstract: Byte Pair Encoding (BPE) constructs vocabularies through greedy pair merging, but the resulting merge order does not necessarily allocate a fixed mod...
arXiv Computation and Language
Oct 9
Language-Specific Effects of Tokenizer Choice in Multilingual Language Models
The study investigates how the choice of tokenizer influences multilingual language models across 54 tokenizers and 123 models. It finds that tokenizer impact is greater for languages with less training data, and that excluding a language from tokenizer training consistently worsens its performance. While reallocating tokenizer training data to lower-resource languages can help, it does not guarantee improvement, and the best tokenizer properties vary by language, enabling predictive screening of tokenizer candidates.
arXiv AI
Sep 28
Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning
The paper introduces TokenAdapt, a model‑agnostic tokenizer transplantation method that uses a hybrid heuristic to initialize new token embeddings, and a novel pre‑tokenization learning approach for multi‑word Supertokens to improve compression. TokenAdapt combines local subword decomposition and global semantic similarity to preserve semantics while reducing retraining needs. Empirical results show that TokenAdapt outperforms existing baselines such as Transtokenizer and ReTok, achieving lower perplexity ratios and significant compression gains.
arXiv Computation and Language
Sep 15
Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories
arXiv:2609.13154v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have made prompts increasingly large and complex. Techniques such as chain-of-thought reasoning (Wei et...