arXiv AI By Haq Nawaz Malik, Nahfid Nissar, Faizan Iqbal

Koshur Diacritizer: A Byte-Level Sequence-to-Sequence Model for Kashmiri Diacritic Restoration

Read the original on arXiv AI →

arXiv:2606. 15883v1 Announce Type: cross Abstract: Kashmiri, an Indo-Aryan language written in a modified Perso-Arabic script, frequently omits diacritic marks in digital text, creating ambiguity and challenging downstream NLP applications.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Aug 20

SuTRA : Structurally-Unified Tokenization with Root Awareness

SuTRA (Structurally-Unified Tokenization with Root Awareness) is a morphology-aware tokenization algorithm designed to address the problem of Morphological Shattering in morphologically rich Indic languages. It preserves the indivisibility of aksharas—complex orthographic syllables—by penalizing merges that cross morphological boundaries, thereby reducing over-fragmentation of words. The authors also release a new morphological segmentation dataset for Hindi, Marathi, and Gujarati, and demonstrate that SuTRA improves morphological alignment by up to 14.7% and semantic recoverability by 34% over BPE, leading to an average machine translation gain of +8.08 chrF2.

By Vaibhav Rathore, Siddhant Gole, Dadhichi Telwadkar, Rooshil Bhatia, Maulik Ruparel, Siddharth Surekha, Neha Bhargava
arXiv Computer Vision
6d ago

A Multi-Stage Framework for Kuzushiji Character Recognition in Japanese Historical Documents

The paper presents a multi‑stage framework for recognizing Kuzushiji characters in Japanese historical documents. It combines character detection, cropping, classification, reading‑order reconstruction via adaptive column clustering, and large‑language‑model‑based post‑OCR correction. The authors also augment data synthetically, correct dataset annotations, and introduce new test sets, achieving significant character error rate reductions on real, synthetic, and out‑of‑domain data.

By Rui-Yang Ju, Kohei Yamashita, Hirotaka Kameko, Shinsuke Mori