← Back to all news
arXiv Computation and Language

5-Dialects-BN: Unmasking the Impact of Transliteration on Bangla Dialectal LLMs

5-Dialects-BN is a new Bangla dialect benchmark that aligns Romanized transliteration with dialectal text, Standard Bangla, English, and subjectivity labels across five regional varieties. The dataset contains 6,000 manually annotated entries from Chittagong, Barisal, Noakhali, Sylhet, and Rangpur, each enriched with five aligned annotations produced and cross‑validated by native speakers and linguistics students. It supports tasks such as dialect identification, normalization, translation, subjectivity classification, and efficient fine‑tuning of multilingual LLMs.

Related stories

arXiv Computation and Language
Sep 22

Cross-Dialect NER for Bangla Regional Dialects Using Leave-One-Dialect-Out Cross-Validation and Explainable AI

The paper introduces a cross-dialect Named Entity Recognition (NER) framework for Bangla, leveraging the ANCHOLIK-NER dataset that covers five major regional dialects. Using a Leave-One-Dialect-Out Cross-Validation strategy, eight transformer-based models were evaluated, with Multilingual-E5 Large achieving the best performance (F1 up to 97.26% on Mymensingh, 82.38% on Chattogram). Local Interpretable Model-agnostic Explanations (LIME) revealed that the models rely mainly on the surface form of entity words rather than surrounding context, suggesting a direction for future improvement.

By Shamim Rahim Refat, Faika Fairuj Preotee, Shuvashis Sarker, Shifat Islam, Bidyarthi Paul, Mohammad Ashraful Hoque
llmsnlpfine-tuningbenchmarkssafety
More like this →
arXiv Computation and Language
Sep 25

COILD: An Indic-Centric Parallel Corpus and Benchmark for Machine Translation Across Indian Languages

COILD is an Indic‑centric parallel corpus that contains over 1.16 million human‑translated and verified sentence pairs across 20 Indian language pairs from four language families. The corpus is sourced from original Indian language materials in eight domains, and a 2,000‑sentence domain‑centric benchmark is provided for consistent multilingual evaluation. Experiments with IndicTrans2‑Distilled and NLLB‑200 show consistent improvements in automatic metrics and human judgments, underscoring the value of high‑quality Indic‑centric data.

By Kshetrimayum Boynao Singh, Nitin Kumar Mishra, Palash Pratim Dutta, Atai Waris Khan, Aparna Kaushik, Avinash Kumar, Deeksha, Deepak Kumar, Saroj Kumar Jha, Saloka Sengupta, Anansa Roy, Umalatha Kannoth, Saifulla Samar, Meena Sharma, Manpreet Kaur, Jyoti Sharma, Ashwini Vaidya, Muralikrishna SN, Md Shad Akhtar, Poonam Bansal, Amita Dev, Sanasam Ranbir Singh, Samit Bhattacharya, Tanmoy Chakraborty, Asif Ekbal
llmsnlpfine-tuningbenchmarks
More like this →
arXiv Computation and Language
Oct 16

Unfragmenting Nawatl NMT: Single-Variant Specialization for Masewalkopa

The paper presents a specialized NMT model for the Masewalkopa variant of Nawatl, a low‑resource language from Mexico. Using a 70,000‑pair multi‑variant foundation, the authors curate 2,665 parallel pairs and augment them with 192 elder‑transcribed sentences via asymmetric back‑translation. A custom finite‑state analyzer reduces unanalyzed forms from 82.0% to 19.9% and supplies functional syntax tags as prefixes, enabling the model to reuse subword embeddings and achieve a dev chrF++ of 58.46 and test chrF++ scores around 58.0. The final LoRA‑v9c‑b model is quantized with CTranslate2 int8, cutting CPU latency from 137.5 to 1.8–2.6 seconds per request for cost‑bounded Cloud Run deployment.

By Luis A. Mercado-Campos, Alexis Palmer
ragfine-tuningefficiency
More like this →
arXiv AI
Sep 3

VakyArth: Evaluating Pragmatic Competence in LLMs across Indic Languages

VakyArth is the first pragmatic benchmark for Indic languages, covering Hindi, Punjabi, Tamil, and Malayalam. It tests models on five pragmatic phenomena—deixis, speech acts, implicature, social pragmatics, and coherence—using multiple-choice questions, natural language inference, and translation tasks authored by native speakers. Evaluation of multilingual LLMs shows consistent failures on pragmatic meanings rooted in Indic linguistic and cultural conventions, with systematic differences across languages and tasks.

By Usneek Singh, Poorvaja Veera Balaji Kumar, Parth Nanda, Anand Madhusoodanan, Geyang Guo, Wei Xu, Junyi Jessy L
llmsbenchmarks
More like this →
arXiv Computation and Language
Sep 1

Generative vs. Encoder Models for Multilingual NER: A Comprehensive Empirical Study on Naamapadam

The paper compares generative and encoder-based neural models for multilingual Named Entity Recognition (NER) across the eleven languages of the Naamapadam benchmark. Five classic model families, four decoder-only large language models fine‑tuned with LoRA and 4‑bit NF4 quantisation, and nine generative models in zero‑to‑5‑shot inference were evaluated under strict CoNLL span‑level metrics. Encoder-based models (mBERT and XLM‑R) achieved substantially higher F1 scores—up to 0.675 on Hindi—than any generative architecture, with gaps of 7.5–40 percentage points; the best few‑shot result reached only 28% of the encoder baseline. The study identifies three language clusters (encoder‑dominant, partial‑coverage, and failure‑zone) and offers deployment guidelines based on transfer learning and low‑resource NLP principles.

By Jakkala Mahesh, Jatavath Shravan Kumar, Komalla Shivani, Sujoy Sarkar
llmsnlpfine-tuningbenchmarks
More like this →