← Back to all news
arXiv Computation and Language By Franck Signe, Hippolyte Pilchen, Fran\c{c}ois Yvon, \'Edouard Grave

To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs

Read the original on arXiv Computation and Language →

The paper proposes a modular tokenizer framework for multilingual large language models, allowing the creation of language‑specific subtokenizers that match monolingual compression quality. It introduces a pretraining strategy that samples these subtokenizers to limit predictions to relevant vocabularies, enabling efficient training and inference. This approach reduces memory usage and speeds up inference without compromising performance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Sep 28

Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning

The paper introduces TokenAdapt, a model‑agnostic tokenizer transplantation method that uses a hybrid heuristic to initialize new token embeddings, and a novel pre‑tokenization learning approach for multi‑word Supertokens to improve compression. TokenAdapt combines local subword decomposition and global semantic similarity to preserve semantics while reducing retraining needs. Empirical results show that TokenAdapt outperforms existing baselines such as Transtokenizer and ReTok, achieving lower perplexity ratios and significant compression gains.

By Shaurya Sharthak, Vinayak Pahalwan, Adithya Kamath, Adarsh Shirawalmath
llmsragnlpfine-tuning
More like this →
arXiv AI
Jul 17

In-Place Tokenizer Expansion for Pre-trained LLMs

arXiv:2607. 15232v1 Announce Type: cross Abstract: A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time.

By Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera, Simon S. Lee, Paul Pak, Aditya Tadimeti, Tim Seyde, Maxime Labonne, Alexander Amini, Mathias Lechner
llmsragnlp
More like this →
arXiv Computation and Language
Oct 9

Latent Core Tokenizer: Compress, but Meaningfully

The paper introduces the Latent Core Tokenizer (LCT), a language‑agnostic method that first discovers reusable linguistic units using Minimum Description Length, entropy‑based boundary signals, and morphotactic constraints before building a shared vocabulary. With a 200K‑token vocabulary across 104 languages, LCT shows lower fertility and higher MorphScore than BPE, Unigram, and parity‑aware BPE, while keeping tokenization cost comparable across languages. On four multilingual downstream benchmarks, LCT outperforms the baselines by 1.48, 1.83, and 2.00 aggregate points, demonstrating that compression alone does not guarantee representation quality and underscoring the role of morphology‑driven structural discovery.

By Felermino D. M. A. Ali, Millicent Ochieng, Ogbemi Ekwejunor-Etchie, Ade Famoti, Jacki O'Neill, Debjit Paul
nlpbenchmarks
More like this →