arXiv Computation and Language By Md Rayhanul Masud, S M Maksudul Alam, Mohammad Jaminur Islam

BANGLAPLAN: A Benchmark for Constraint-Based Planning and Decision-Making in Bangla

Read the original on arXiv Computation and Language →

BANGLAPLAN is a new benchmark comprising 1,000 Bangla planning and decision‑making problems across ten everyday task families, presented in four language registers (everyday, formal, colloquial, and Bangla–English mixed). The benchmark includes both feasible and infeasible instances and requires answers in eight structured JSON formats such as schedules, routes, and allocations. Evaluation of 11 language‑model configurations shows a wide performance gap, with GPT‑6.1 Sol achieving 98.7% accuracy while most models perform poorly, highlighting the challenge of constraint‑based planning in Bangla.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Hugging Face Trending Papers
Jul 14

GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models

The evaluation of mathematical reasoning in large language models (LLMs) has predominantly focused on high-resource languages like English. This has created a significant barrier to the equitable development and deployment of AI in linguistically diverse regions such as Bangladesh, where over 230 million people speak Bengali.

arXiv Computation and Language
Sep 22

CCTU: A Benchmark for Tool Use under Complex Constraints

CCTU is a new benchmark designed to evaluate large language models (LLMs) on their ability to use tools under complex constraints. It includes 200 test cases that average seven constraint types and 4,700‑token prompts, covering resource, behavior, toolset, and response dimensions. An executable validation module performs step‑level checks, and nine state‑of‑the‑art LLMs were tested, revealing that none exceed a 20% task completion rate when strict constraints are enforced, with frequent violations and limited self‑refinement.

By Junjie Ye, Guoqiang Zhang, Wenjie Fu, Zelin Li, Tao Gui, Qi Zhang, Xuanjing Huang
arXiv AI
Jun 2

TravelEval: A Comprehensive Benchmarking Framework for Evaluating LLM-Powered Travel Planning Agents

arXiv:2606. 01046v1 Announce Type: new Abstract: The development of Large Language Models (LLMs) has significantly improved travel planning applications, yet evaluating such models is limited by existing benchmarks' limitations: 1) overemphasis on constraint compliance, neglecting multi-dimensional qualities like spatio-temporal cost; 2) datasets lacking real-world authenticity and coverage in key areas (e.

By Weiyi Chen, Shuaixiong Wang, Ziyun Gao, Kaichun Hu, Wangze Ni, Shimin Di, Chen Jason Zhang, Lei Chen