arXiv AI

Bangla Sentence Function Classification: Corpus Development, Model Benchmarking, and Interpretability

arXiv Computation and Language
1d ago

Cross-Dialect NER for Bangla Regional Dialects Using Leave-One-Dialect-Out Cross-Validation and Explainable AI

The paper introduces a cross-dialect Named Entity Recognition (NER) framework for Bangla, leveraging the ANCHOLIK-NER dataset that covers five major regional dialects. Using a Leave-One-Dialect-Out Cross-Validation strategy, eight transformer-based models were evaluated, with Multilingual-E5 Large achieving the best performance (F1 up to 97.26% on Mymensingh, 82.38% on Chattogram). Local Interpretable Model-agnostic Explanations (LIME) revealed that the models rely mainly on the surface form of entity words rather than surrounding context, suggesting a direction for future improvement.

By Shamim Rahim Refat, Faika Fairuj Preotee, Shuvashis Sarker, Shifat Islam, Bidyarthi Paul, Mohammad Ashraful Hoque
arXiv Computation and Language
Sep 4

To What Extent Do Large Language Models Understand Bangla Idioms?

The paper introduces the first large‑scale benchmark dataset of Bangla idioms, along with a synthetic multiple‑choice question set for idiom meaning identification. It evaluates recent large language models on three idiom‑related tasks—paraphrasing, idiom span detection, and meaning identification—using zero‑shot and few‑shot prompting. Results show significant variability across models, with Phi‑4‑mini‑instruct best at paraphrasing, Kimi‑K2‑32b‑instruct excelling at span detection, and Gemini‑2.5‑flash leading in meaning identification.

By Mousumi Akter, Md. Faiyaz Abdullah Sayeedi, Nurul Labib Sayeedi, Swakkhar Shatabda
arXiv Computation and Language
Sep 3

TalkFa: A Unified Benchmark for Farsi Dialogue Generation and Understanding

TalkFa introduces a unified benchmark for Farsi dialogue generation and understanding, comprising three datasets: WIKI‑FADIAL (4.2K Wikipedia‑grounded dialogues), DAILYDIALOG‑FA (6.6K dialogues with dialogue‑act and emotion annotations), and PLAYDIAL‑FA (2.1K theatrical dialogues with sentiment labels). All dialogues are curated through multi‑stage review by native speakers, ensuring high quality. Experiments show that LoRA fine‑tuning improves generation performance with less data, while specific models excel on classification tasks, and human evaluation confirms the benchmark’s reliability.

By Neda Jamshidi, Kamyar Zeinalipour, Fahimeh Akbari, Monica Bianchini, Marco Maggini, Marco Gori
arXiv Machine Learning
Jul 31

Selecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 Models

arXiv:2607. 27421v1 Announce Type: cross Abstract: Intent classification is a core component of task-oriented dialogue systems, yet practitioners have limited systematic guidance for selecting deployable open-weight language models under compute, latency, and robustness constraints.

By Parishruthi Ganesh, Gerry Dozier, Cheryl Seals
arXiv AI
Jun 3

Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling

arXiv:2606. 02837v1 Announce Type: cross Abstract: Accurate translation from Natural Language to First-Order Logic (NL-to-FOL) underpins neurosymbolic AI systems and Natural Language Inference (NLI), making the quality of NL-to-FOL benchmarks essential -- yet these datasets have never been rigorously audited.

By Andrea Brunello, Cristian Curaba, Luca Geatti, Michele Mignani, Angelo Montanari, Nicola Saccomanno
arXiv Computation and Language
Sep 4

The Impact of Synthetic Data Augmentation on Discourse-Pragmatic Function Classification

The study examines how the geometric placement of synthetic data affects discourse‑pragmatic function classification. Using 410 annotated instances of the word "look" from the British National Corpus, synthetic examples were generated with Llama 3.1 and grouped by cosine distance from real data in RoBERTa space. Six training conditions were compared, showing that examples close to real data (NEAR) yield the largest macro‑F gain, while a distance‑balanced mix gives the highest accuracy, yet none improve AUC.

By Sara Sorahi, Kevin Tang, Reza Kazemian
arXiv Machine Learning
Sep 4

Unifying Conformal Language Tasks with In-Context Ensembles

The paper introduces the Conformal Relevance framework, which employs in-context learning example curation and ensembling to generate a score function that preserves coverage while enhancing conciseness for NLP tasks such as summarization and extractive question answering. Unlike previous methods that rely on labor-intensive, task‑specific prompt engineering, this approach requires minimal manual input. The authors validate the framework across seven NLP tasks and provide a theoretical analysis of how diversity in ensembled conformal scores can improve worst‑case sentence scores, including a saturation bound on ensemble gains.

By Xiao Shi Huang, Chen-Yuan Lin, Bruce Kuwahara, Kin Kwan Leung, Jesse C. Cresswell