arXiv:2606. 29243v2 Announce Type: replace Abstract: We introduce KrishokChat, an 85,979-instance Bengali agricultural benchmark built from 284 government publications, 13 institutions, and six regional dialects.
arXiv:2606. 29243v1 Announce Type: new Abstract: We present KrishokChat, the first citation-grounded Bengali agricultural instruction-tuning dataset for crop advisory in low-resource settings.
arXiv:2606. 28992v1 Announce Type: cross Abstract: General-purpose large language models (LLMs) have demonstrated strong abilities in opendomain question answering, information extraction, and text generation.
The paper introduces BanglaSafe, a benchmark of 879 Bengali prompts that covers 17 culturally grounded harm categories and five prompting conditions. Evaluation of 18 frontier LLMs shows that 53.6% of responses are unsafe or partially unsafe, with 14.7% containing strictly harmful content. The study finds that the writing style within Bengali has a stronger impact on safety than the language switch itself, and that current safety classifiers struggle to reliably evaluate Bengali content.
arXiv:2605. 31483v1 Announce Type: cross Abstract: Despite Bengali being the sixth most spoken language in the world, no prior work has systematically evaluated hallucination in large language models (LLMs) for Bengali.
Safety benchmarks for large language models often assess the risk of a user query, although the outcome of question answering depends on whether the response violates a policy. This distinction is cri...
IndicBankBench is a 799‑case benchmark designed to evaluate the safety and reliability of language model assistants in Indian retail banking. It covers five operational domains, a capability/refusal domain, and twenty primary axes, assessing each case at four stages: safety, action and tool use, response adequacy, and advisory quality. The benchmark uses deterministic safety checks, a narrow resolver for ambiguous confirmation‑before‑write scenarios, and an LLM judge for semantic response adequacy, reporting strict pass rates that reveal a gap between strict reliability (43.7%–58.2%) and at‑least‑once success (60%–74%).
arXiv:2609.01210v1 Announce Type: cross Abstract: Safety benchmarks for large language models often assess the risk of a user query, although the outcome of question answering depends on whether the...
arXiv:2608.21880v1 Announce Type: new Abstract: Bangla large language model (LLM) safety is difficult to evaluate with English-centric or standard-script benchmarks because Bangla users routinely wri...
IndicSafeEval is a new evaluation framework that tests the safety robustness of large language models against persuasion-based jailbreak attacks in Indian languages. The benchmark includes 7,200 adversarial prompts covering ten safety-critical content categories, six persuasive strategies, and four languages (Hindi, Bengali, Marathi, Punjabi). Experiments show that model safety varies significantly across languages, prompt styles, and risk categories, revealing that current English-centric safety evaluations miss important multilingual vulnerabilities.
The paper introduces SEAV, a verification‑centric framework for evaluating jailbreak attempts against large language models. SEAV decomposes responses into ordered steps and checks both validity and correctness using LLM‑as‑a‑judge and retrieval‑grounded verification. The method reduces false positives by 14.9 percentage points on a strategic‑dishonesty diagnostic and reclassifies 22.1–51.0% of previously successful jailbreaks as invalid across multiple benchmarks.
KhatianDoc is a new benchmark that tests multimodal large language models on Bengali legal land records, specifically the handwritten RS Khatians used in Bangladesh. The benchmark comprises four tasks—symbol recognition, base‑16 to decimal conversion, structured field extraction, and legal document question answering—drawn from 107 real records and 1,634 QA pairs. Six multimodal LLMs were evaluated under a zero‑shot protocol, revealing that many models fail to answer a significant portion of questions correctly and perform poorly on arithmetic conversion, highlighting a lack of capability rather than a performance gap.
IndicSafeEval is a new evaluation framework that tests the safety robustness of large language models against persuasion-based jailbreak attacks in Indian languages. The benchmark covers ten safety-critical content categories, six persuasive strategies, and four languages—Hindi, Bengali, Marathi, and Punjabi—producing 7,200 adversarial prompts. Experiments show that model safety varies significantly across languages, prompt styles, and risk categories, highlighting gaps in current English-centric safety assessments.