The paper compares generative and encoder-based neural models for multilingual Named Entity Recognition (NER) across the eleven languages of the Naamapadam benchmark. Five classic model families, four decoder-only large language models fine‑tuned with LoRA and 4‑bit NF4 quantisation, and nine generative models in zero‑to‑5‑shot inference were evaluated under strict CoNLL span‑level metrics. Encoder-based models (mBERT and XLM‑R) achieved substantially higher F1 scores—up to 0.675 on Hindi—than any generative architecture, with gaps of 7.5–40 percentage points; the best few‑shot result reached only 28% of the encoder baseline. The study identifies three language clusters (encoder‑dominant, partial‑coverage, and failure‑zone) and offers deployment guidelines based on transfer learning and low‑resource NLP principles.
By Jakkala Mahesh, Jatavath Shravan Kumar, Komalla Shivani, Sujoy Sarkar
arXiv:2607. 23319v1 Announce Type: cross Abstract: Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages.
By Poornima Kumaresan, Pavithra Muruganantham, Lakshmi Rajendran, Santhosh Sivasubramani
arXiv:2606. 24825v1 Announce Type: cross Abstract: Part-of-Speech (POS) tagging is a foundational NLP task underpinning machine translation, information extraction, and syntactic parsing.
By Hariom Ingle, Ronit Ghode, Ishwari Gondkar, Jidnyasa Harad, Raviraj Joshi
arXiv:2607. 24276v1 Announce Type: cross Abstract: Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words.
By Priyansh Srivastava
We present DONDO, a family of open, permissively licensed automatic speech recognition (ASR) base models for African languages, built on the w2v-BERT 2. 0 self-supervised speech encoder.
Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words. Because these tokenizers are trained predominantly on English-centric corpora, they introduce a systematic and often overlooked disadvantage for many non-English languages.
arXiv:2607. 23344v1 Announce Type: cross Abstract: Named Entity Recognition (NER) for low-resource languages such as Marathi remains a challenging task due to limited annotated resources and linguistic complexity.
By Hariom Ingle, Ronit Ghode, Ishwari Gondkar, Jidnyasa Harad, Raviraj Joshi
COILD is an Indic‑centric parallel corpus that contains over 1.16 million human‑translated and verified sentence pairs across 20 Indian language pairs from four language families. The corpus is sourced from original Indian language materials in eight domains, and a 2,000‑sentence domain‑centric benchmark is provided for consistent multilingual evaluation. Experiments with IndicTrans2‑Distilled and NLLB‑200 show consistent improvements in automatic metrics and human judgments, underscoring the value of high‑quality Indic‑centric data.
By Kshetrimayum Boynao Singh, Nitin Kumar Mishra, Palash Pratim Dutta, Atai Waris Khan, Aparna Kaushik, Avinash Kumar, Deeksha, Deepak Kumar, Saroj Kumar Jha, Saloka Sengupta, Anansa Roy, Umalatha Kannoth, Saifulla Samar, Meena Sharma, Manpreet Kaur, Jyoti Sharma, Ashwini Vaidya, Muralikrishna SN, Md Shad Akhtar, Poonam Bansal, Amita Dev, Sanasam Ranbir Singh, Samit Bhattacharya, Tanmoy Chakraborty, Asif Ekbal
arXiv:2610.19388v1 Announce Type: new
Abstract: Massively multilingual speech modeling has made remarkable progress in recent years, yet existing training resources and benchmarks still underrepresen...
By Sangmin Lee, Yejin Jeon, David Ifeoluwa Adelani
arXiv:2610.20279v1 Announce Type: new
Abstract: Part-of-speech tagging is a fundamental sequence labeling task and an important component for many downstream natural language processing applications....
By Usman Nawaz, Muhammad Junaid Iqbal, Muhammad Asaf, Mariam Munsif Mir, Faisal Rehman, Tahir Alyas, Muhammad Hasnain
NEPSEE is a Nepali sentiment analysis benchmark created by having two annotators independently label 5,614 tweets from Nepali public figures on a five‑point scale. Only 3,189 sentences with identical polarity are kept for a three‑class version, while 766 sentences with adjacent labels are used in a five‑point task. Baseline results show NepBERTa achieving a macro‑F1 of 0.877 on the three‑class benchmark, with commercial LLMs performing similarly; on the five‑point task, Claude Opus 5.5 outperforms encoders by 0.10–0.12 macro‑F1, and a tokenizer audit reveals that restoring Devanagari vowel signs can significantly alter performance.
By Gaurab Baral, Mimansak Nepal, Grishma Basnet
IndicTriMix presents a new benchmark and models for token‑level language identification in tri‑language code‑mixed text involving Hindi, Gujarati, and Bengali. The authors reformulate the task as sequence labeling and fine‑tune transformer models MuRIL and XLM‑RoBERTa, evaluating them on manually annotated test sets. They also introduce two code‑mixing generation methods using parallel sentences and release the datasets and fine‑tuned models for public use.
By Pruthwik Mishra, Rudra Trivedi, Avi Patel, Ashok Urlana, Shrikant Malviya