The paper introduces UniLID, a lightweight language identification method that uses the UnigramLM tokenization algorithm to predict a string’s language by evaluating which language’s unigram distribution best explains the text. UniLID is data‑ and compute‑efficient, allows incremental addition of new languages without retraining, and can be integrated into existing tokenization pipelines. Experiments show competitive performance against baselines such as fasttext, GlotLID‑M, and CLD3, achieving 69% accuracy with five labeled samples per language and 89% with 25, and delivering significant gains on fine‑grained dialect identification.
arXiv:2606.23566v3 Announce Type: replace Abstract: Language-specific tokenizers improve tokenization quality and the downstream performance of models on those languages. Using one with an existing p...
arXiv:2610.16601v1 Announce Type: cross Abstract: Processing multilingual text requires both identifying a broad range of languages and distinguishing the language of each token in inputs containing...
arXiv:2512. 20757v2 Announce Type: replace-cross Abstract: Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs).
arXiv:2510.18019v3 Announce Type: replace Abstract: Multilingual watermarking aims to make large language model (LLM) outputs traceable across languages, yet current methods still fall short. Despite...
IndicTriMix presents a new benchmark and models for token‑level language identification in tri‑language code‑mixed text involving Hindi, Gujarati, and Bengali. The authors reformulate the task as sequence labeling and fine‑tune transformer models MuRIL and XLM‑RoBERTa, evaluating them on manually annotated test sets. They also introduce two code‑mixing generation methods using parallel sentences and release the datasets and fine‑tuned models for public use.
The paper introduces the Latent Core Tokenizer (LCT), a language‑agnostic method that first discovers reusable linguistic units using Minimum Description Length, entropy‑based boundary signals, and morphotactic constraints before building a shared vocabulary. With a 200K‑token vocabulary across 104 languages, LCT shows lower fertility and higher MorphScore than BPE, Unigram, and parity‑aware BPE, while keeping tokenization cost comparable across languages. On four multilingual downstream benchmarks, LCT outperforms the baselines by 1.48, 1.83, and 2.00 aggregate points, demonstrating that compression alone does not guarantee representation quality and underscoring the role of morphology‑driven structural discovery.
The paper investigates whether intrinsic tokenizer features can predict the performance of downstream language models. By training about 200 models across nine languages and evaluating numerous tokenizer metrics, the authors find that none of the existing metrics consistently forecasts model performance across different families, tasks, and languages. They propose a new linguistically grounded metric that captures aspects of tokenizer quality missed by prior measures, though its predictive power may still vary.
The paper proposes a modular tokenizer framework for multilingual large language models, allowing the creation of language‑specific subtokenizers that match monolingual compression quality. It introduces a pretraining strategy that samples these subtokenizers to limit predictions to relevant vocabularies, enabling efficient training and inference. This approach reduces memory usage and speeds up inference without compromising performance.
arXiv:2506. 15138v2 Announce Type: replace-cross Abstract: Tokenization directly affects the inference efficiency of large language models, since fragmented tokenization increases sequence length and generation cost.
arXiv:2608. 07629v1 Announce Type: cross Abstract: Multilingual neural machine translation models such as NLLB-200 cover 200 languages but leave thousands unsupported, including most Grassfields Bantu languages of Cameroon.
arXiv:2607. 23319v1 Announce Type: cross Abstract: Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages.
arXiv:2610.33017v1 Announce Type: new Abstract: Multilingual tokenizers trained on English-dominated corpora often over-segment Indic languages, resulting in longer token sequences and increased toke...