arXiv Computation and Language
Sep 22

Type-Driven Tokenization for Brahmic Scripts

arXiv:2609.22125v1 Announce Type: new Abstract: Standard tokenizers used in large language models produce malformed text when applied to Brahmic scripts. They are a family of abugidas, writing system...

By Sai Hemanth Kapila, Rakshika Bagavathy