arXiv:2606. 15144v1 Announce Type: cross Abstract: Large language models (LLMs) process text as sequences of subword tokens, which can obscure the character-level and morphological structure that underlies word formation.
By Jann Railey Montalan, David Demitri Africa, Jimson Paulo Layacan, Richell Isaiah Flores, Ivan Yuri De Leon, Lance Calvin Gamboa
MoVoC introduces a morphology-aware subword construction method for Ge'ez script languages, integrating supervised morphological analysis with Byte Pair Encoding to preserve morphological boundaries while retaining lexical meaning. The resulting MoVoC-Tok tokenizer outperforms baselines in MorphScore and morpheme boundary precision across several low-resource, morphologically complex languages, notably achieving significant gains on Tigre through cross-lingual application. The authors also release morpheme-annotated datasets and a morpheme-aware vocabulary for public use.
By Hailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang Nejdl
arXiv:2607. 23319v1 Announce Type: cross Abstract: Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages.
By Poornima Kumaresan, Pavithra Muruganantham, Lakshmi Rajendran, Santhosh Sivasubramani
The paper shows that HuggingFace’s ByteLevel pre‑tokenizer, which treats a word as a sequence of Unicode letters, splits abugida scripts at every vowel sign, creating a training‑free lower bound on tokenizer fertility. Across 26 languages, all 17 abugidas exhibit increased token counts (up to 9×), while Latin, Cyrillic, Hangul, and Han remain unchanged. The authors demonstrate that correcting the character class reduces Nepali token counts, improves model performance, and that this issue is widespread in popular HuggingFace models.
By Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami Pun
The paper investigates Japanese past‑tense verb inflection in neural morphological generation systems, focusing on a rare irregular subtype where stems end in /e/ and require gemination before the suffix. Despite overall high accuracy (>97%), this <1% subclass accounts for 30–43% of residual errors, amplifying its impact by 34–48 times its frequency. Ablation experiments show that removing this specific subtype yields larger accuracy gains than removing all irregular verbs, highlighting the role of low‑frequency patterns and orthographic processes in error concentration.
By Wen Zhang
The article examines how Byte‑Pair Encoding (BPE) tokenization handles Polish, an inflectional language, and finds that BPE tends to stabilize frequent surface fragments of grammatical exponents rather than true grammatical categories. It introduces the concept of grammatical form anchoring, showing that certain Polish verb forms can signal the speaking subject without an explicit pronoun, and highlights that language models may lack a stable grammatical "I" and can shift gender or mirror user forms. The study proposes Roclawski’s segmentation‑flexional forms as a diagnostic framework and suggests that more stable Polish modeling would require sublexical stabilization, anchoring grammatical form in the inflectional system, and maintaining the grammatical "I" in dialogue.
By Elzbieta Dawidek (University of Lower Silesia DSW Ideis)