arXiv Computation and Language October 23, 2026 By Hailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang Nejdl

MoVoC: Morphology-Aware Subword Construction for Geez Script Languages Corrected version

Read the original on arXiv Computation and Language →

MoVoC introduces a morphology-aware subword construction method for Ge'ez script languages, integrating supervised morphological analysis with Byte Pair Encoding to preserve morphological boundaries while retaining lexical meaning. The resulting MoVoC-Tok tokenizer outperforms baselines in MorphScore and morpheme boundary precision across several low-resource, morphologically complex languages, notably achieving significant gains on Tigre through cross-lingual application. The authors also release morpheme-annotated datasets and a morpheme-aware vocabulary for public use.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Jul 28

BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis

arXiv:2607. 23319v1 Announce Type: cross Abstract: Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages.

By Poornima Kumaresan, Pavithra Muruganantham, Lakshmi Rajendran, Santhosh Sivasubramani

llms nlp benchmarks

arXiv Machine Learning

Oct 16

Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog

The study investigates whether morphological correctness influences tokenization effectiveness in Tagalog, a language with infix morphology that disrupts contiguous roots. Researchers developed MorphAustro‑Tok, a tokenizer that preserves the root as a contiguous unit and separates infixes, and compared it to a standard BPE baseline and a scrambled‑morpheme control that randomizes infix placement. Across multiple runs, MorphAustro‑Tok consistently outperformed the baseline, but the scrambled control showed no significant difference, indicating that morphological correctness does not contribute to the observed gains at this scale.

By Abdul Rehman Antall, Baidya Nath Saha

computer-vision nlp

arXiv Computation and Language

5d ago

Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya

arXiv:2607.15209v4 Announce Type: replace Abstract: Multilingual pre-trained language models such as XLM-R perform well for major languages but struggle with low-resource Ge'ez-script languages, larg...

By Hailay Kidu Teklehaymanot, Debela Desalegn Yadeta, Wolfgang Nejdl

llms rag nlp fine-tuning benchmarks

arXiv AI

Jul 10

Less Is More: Reducing Token Counts Without Compromising Performance

arXiv:2506. 15138v2 Announce Type: replace-cross Abstract: Tokenization directly affects the inference efficiency of large language models, since fragmented tokenization increases sequence length and generation cost.

By Gyeongje Cho, Yeonkyoung So, Sangmin Lee, Jaejin Lee

llms nlp

arXiv Computation and Language

Sep 16

The Functionalizer: Lossless Functional Decomposition for Subword Tokenization

arXiv:2609.15991v1 Announce Type: new Abstract: Standard subword tokenizers either treat every orthographic variation of a word (such as hello, Hello, HELLO, and H\'ello) as unrelated vocabulary entr...

By Connor Makowski, Willem Guter

llms rag nlp

arXiv AI

Jun 18

Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish

arXiv:2606. 18717v1 Announce Type: cross Abstract: Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text.

By Tolga \c{S}akar

llms rag nlp safety