arXiv Computation and Language
A Character-Level Neural Approach to Sinhala Sandhi Splitting
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Computation and Language
Sep 21
Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining
arXiv:2609. 21362v1 Announce Type: new Abstract: Conventional tokenizers represent text as characters or statistically derived subwords, overlooking the internal phonological structure of syllables and often requiring large vocabularies.
arXiv Computation and Language
Sep 4
Contextual Tamil Spelling and Grammar Correction Using Progressively Fine-Tuned Sequence-to-Sequence Transformers
The paper presents an end‑to‑end sequence‑to‑sequence approach for correcting Tamil spelling and grammar errors, leveraging progressively fine‑tuned transformer models (mT5‑small and mBART‑50). Using a synthetic corpus of 657,720 noisy‑clean sentence pairs across ten error categories, the authors introduce a four‑stage training schedule that targets surface noise, contextual grammar, single‑site sandhi, and multi‑site cross‑word sandhi. The best model, mBART‑50 v5, achieves 69.3% exact‑match accuracy on a balanced diagnostic set, with notable gains in sandhi (87.5%) and subject‑verb agreement (43.5%) accuracy, while also revealing a precision‑recall trade‑off for sandhi corrections.
arXiv Computation and Language
Sep 1
SinLlama -- A Large Language Model for Sinhala
The paper introduces SinLlama, the first decoder‑based open‑source large language model with explicit support for Sinhala. By extending Llama‑3‑8B, adding Sinhala‑specific tokenizer vocabulary, and performing continual pre‑training on a cleaned 10‑million‑token Sinhala corpus, the authors created a model that surpasses both the base and instruction‑fine‑tuned variants of Llama‑3‑8B on three text classification tasks. This work addresses the underrepresentation of low‑resource languages in open‑source LLMs.
arXiv AI
Jun 16
PACUTE: Phonology-, Affix-, and Character-level Understanding of Tokens for Filipino
arXiv:2606. 15144v1 Announce Type: cross Abstract: Large language models (LLMs) process text as sequences of subword tokens, which can obscure the character-level and morphological structure that underlies word formation.
arXiv Machine Learning
Aug 4
Latent Softmax for Data-Efficient Phoneme-Based Multilingual ASR Across Tonal and Non-Tonal Languages
arXiv:2608. 01281v1 Announce Type: cross Abstract: Phoneme-based multilingual automatic speech recognition (ASR) can share acoustic evidence across languages more directly than language-specific subword modeling.
arXiv AI
Jun 18
Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
arXiv:2606. 18717v1 Announce Type: cross Abstract: Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text.