MoVoC introduces a morphology-aware subword construction method for Ge'ez script languages, integrating supervised morphological analysis with Byte Pair Encoding to preserve morphological boundaries while retaining lexical meaning. The resulting MoVoC-Tok tokenizer outperforms baselines in MorphScore and morpheme boundary precision across several low-resource, morphologically complex languages, notably achieving significant gains on Tigre through cross-lingual application. The authors also release morpheme-annotated datasets and a morpheme-aware vocabulary for public use.
arXiv:2607. 23319v1 Announce Type: cross Abstract: Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages.
The study investigates whether morphological correctness influences tokenization effectiveness in Tagalog, a language with infix morphology that disrupts contiguous roots. Researchers developed MorphAustro‑Tok, a tokenizer that preserves the root as a contiguous unit and separates infixes, and compared it to a standard BPE baseline and a scrambled‑morpheme control that randomizes infix placement. Across multiple runs, MorphAustro‑Tok consistently outperformed the baseline, but the scrambled control showed no significant difference, indicating that morphological correctness does not contribute to the observed gains at this scale.
arXiv:2607.15209v4 Announce Type: replace Abstract: Multilingual pre-trained language models such as XLM-R perform well for major languages but struggle with low-resource Ge'ez-script languages, larg...
arXiv:2506. 15138v2 Announce Type: replace-cross Abstract: Tokenization directly affects the inference efficiency of large language models, since fragmented tokenization increases sequence length and generation cost.
arXiv:2609.15991v1 Announce Type: new Abstract: Standard subword tokenizers either treat every orthographic variation of a word (such as hello, Hello, HELLO, and H\'ello) as unrelated vocabulary entr...
arXiv:2606. 18717v1 Announce Type: cross Abstract: Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text.
The paper introduces the Latent Core Tokenizer (LCT), a language‑agnostic method that first discovers reusable linguistic units using Minimum Description Length, entropy‑based boundary signals, and morphotactic constraints before building a shared vocabulary. With a 200K‑token vocabulary across 104 languages, LCT shows lower fertility and higher MorphScore than BPE, Unigram, and parity‑aware BPE, while keeping tokenization cost comparable across languages. On four multilingual downstream benchmarks, LCT outperforms the baselines by 1.48, 1.83, and 2.00 aggregate points, demonstrating that compression alone does not guarantee representation quality and underscoring the role of morphology‑driven structural discovery.
CWoMP (Contrastive Word‑Morpheme Pretraining) is a new approach for generating interlinear glossed text that treats morphemes as atomic form‑meaning units with learned representations. It uses a contrastively trained encoder to align words in context with their constituent morphemes in a shared embedding space, and an autoregressive decoder that retrieves morpheme sequences from a mutable lexicon of these embeddings. The method yields interpretable predictions grounded in lexicon entries and allows users to improve results at inference time by expanding the lexicon without retraining, achieving superior performance and efficiency on diverse low‑resource languages, especially in extremely low‑resource settings.
arXiv:2610.08794v1 Announce Type: new Abstract: Large language models rely on subword tokenizers whose quality varies across languages, yet no standardized multi-metric framework exists for broad com...
arXiv:2608.27658v1 Announce Type: new Abstract: Subword tokenization hinders low-resource language processing by imposing frequency patterns from dominant languages onto script-sharing variants. Byte...
The paper shows that HuggingFace’s ByteLevel pre‑tokenizer, which treats a word as a sequence of Unicode letters, splits abugida scripts at every vowel sign, creating a training‑free lower bound on tokenizer fertility. Across 26 languages, all 17 abugidas exhibit increased token counts (up to 9×), while Latin, Cyrillic, Hangul, and Han remain unchanged. The authors demonstrate that correcting the character class reduces Nepali token counts, improves model performance, and that this issue is widespread in popular HuggingFace models.
Pearl-Chat is an 8.5‑million‑parameter causal decoder‑only transformer built from first principles in JAX and Flax NNX, specifically engineered for the low‑resource Bantu language Luganda. It uses a byte‑level BPE tokenizer trained from scratch on curated Luganda corpora, achieving a two‑fold compression improvement over larger models like LLaMA‑3 and Gemma. The model is trained in two stages—pre‑training on diverse Luganda text and instruction fine‑tuning on dialogues—reaching a validation perplexity of 1.20 and high ROUGE‑L and BLEU scores on a held‑out diagnostic benchmark, with all weights, tokenizers, and code released openly.