arXiv Computation and Language
LangMAP: A Language-Adaptive Approach to Tokenization
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Computation and Language
Sep 15
To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs
The paper proposes a modular tokenizer framework for multilingual large language models, allowing the creation of language‑specific subtokenizers that match monolingual compression quality. It introduces a pretraining strategy that samples these subtokenizers to limit predictions to relevant vocabularies, enabling efficient training and inference. This approach reduces memory usage and speeds up inference without compromising performance.
arXiv AI
Sep 28
Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning
The paper introduces TokenAdapt, a model‑agnostic tokenizer transplantation method that uses a hybrid heuristic to initialize new token embeddings, and a novel pre‑tokenization learning approach for multi‑word Supertokens to improve compression. TokenAdapt combines local subword decomposition and global semantic similarity to preserve semantics while reducing retraining needs. Empirical results show that TokenAdapt outperforms existing baselines such as Transtokenizer and ReTok, achieving lower perplexity ratios and significant compression gains.
arXiv Machine Learning
Jul 7
TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
arXiv:2512. 20757v2 Announce Type: replace-cross Abstract: Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs).
Hugging Face Trending Papers
Sep 14
To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs
Multilingual Large Language Models (LLMs) traditionally rely on a single vocabulary shared by all supported languages, which can lead to uneven compression across them. Moreover, their large embedding...
arXiv Computation and Language
Oct 16
No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance
The paper investigates whether intrinsic tokenizer features can predict the performance of downstream language models. By training about 200 models across nine languages and evaluating numerous tokenizer metrics, the authors find that none of the existing metrics consistently forecasts model performance across different families, tasks, and languages. They propose a new linguistically grounded metric that captures aspects of tokenizer quality missed by prior measures, though its predictive power may still vary.
arXiv Machine Learning
Jun 16
Emergent retokenization symmetry in large language models: phenomenology and applications
arXiv:2606. 15521v1 Announce Type: cross Abstract: Tokenization introduces representational redundancy: under a fixed token vocabulary, every byte string admits many valid token encodings, or segmentations, that decode to the same surface string.