The Flow has not summarised this story yet — read it at arXiv Computation and Language.
arXiv:2606. 24825v1 Announce Type: cross Abstract: Part-of-Speech (POS) tagging is a foundational NLP task underpinning machine translation, information extraction, and syntactic parsing.
arXiv:2607. 23319v1 Announce Type: cross Abstract: Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages.
arXiv:2609. 21362v1 Announce Type: new Abstract: Conventional tokenizers represent text as characters or statistically derived subwords, overlooking the internal phonological structure of syllables and often requiring large vocabularies.
arXiv:2608.23448v1 Announce Type: new Abstract: This paper presents new Cantonese ParGram resources and evaluates LLMs for knowledge-driven grammar engineering within a controlled experimental paradi...
5-Dialects-BN is a new Bangla dialect benchmark that aligns Romanized transliteration with dialectal text, Standard Bangla, English, and subjectivity labels across five regional varieties. The dataset contains 6,000 manually annotated entries from Chittagong, Barisal, Noakhali, Sylhet, and Rangpur, each enriched with five aligned annotations produced and cross‑validated by native speakers and linguistics students. It supports tasks such as dialect identification, normalization, translation, subjectivity classification, and efficient fine‑tuning of multilingual LLMs.
SuTRA (Structurally-Unified Tokenization with Root Awareness) is a morphology-aware tokenization algorithm designed to address the problem of Morphological Shattering in morphologically rich Indic languages. It preserves the indivisibility of aksharas—complex orthographic syllables—by penalizing merges that cross morphological boundaries, thereby reducing over-fragmentation of words. The authors also release a new morphological segmentation dataset for Hindi, Marathi, and Gujarati, and demonstrate that SuTRA improves morphological alignment by up to 14.7% and semantic recoverability by 34% over BPE, leading to an average machine translation gain of +8.08 chrF2.