arXiv Computation and Language By Shrenil Shaun Sharma, Avi Sharma

SCHEDBench: A Benchmark for Evaluating LLM Constraint Faithfulness in Natural-Language Combinatorial Scheduling

Read the original on arXiv Computation and Language →

SCHEDBench is a natural‑language benchmark that evaluates whether large language models (LLMs) produce schedules that remain constraint‑feasible when the same scheduling problem is expressed in different natural‑language surface forms. The benchmark covers 1,132 instances from job‑shop scheduling, resource‑constrained project scheduling, nurse rostering, and curriculum timetabling, and uses domain‑specific templates and surface‑form variations to generate varied problem statements. Experiments with thirteen frontier and open‑weight LLMs show that models are not reliably invariant to semantically equivalent renderings, with surface‑form variation reducing feasibility and increasing hard‑constraint violations, especially when constraints are reordered.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI
Aug 20

Improving Natural-Language Combinatorial-Optimization Accuracy in Resource-Constrained Language Models via Formal Abstractions

The paper introduces SDDL, a neuro‑symbolic framework that converts natural‑language combinatorial scheduling problems into compact, solver‑aligned representations, delegating low‑level modeling and search to a deterministic compiler and external solver. On a 300‑instance subset of scheduling tasks, SDDL achieves higher feasibility rates for resource‑constrained language models—up to 55.3% and 28.3%—compared to direct‑generation baselines (23.7% and 1.3%) and solver‑code baselines (21.7% and 7.0%), with a median optimality gap of 0.0% among feasible schedules.

By Shrenil Shaun Sharma, Avi Sharma
arXiv AI
Oct 19

LangNCO: Compiling Natural-Language Constraints into Inference-Time Interventions for Neural Combinatorial Optimization

LangNCO is a framework that transforms natural-language constraints into structured inference-time interventions for a frozen neural combinatorial optimization (NCO) solver. It uses a pretrained large language model to convert each requirement into a domain-specific language rule set, which a deterministic Rule Executor applies as hard action masks. Critical-Offset Decoding then reallocates rollout budget to ensure feasibility, achieving near-perfect feasibility across multiple constraint types without retraining the solver.

By Hironori Ohigashi
arXiv Computation and Language
Sep 22

CCTU: A Benchmark for Tool Use under Complex Constraints

CCTU is a new benchmark designed to evaluate large language models (LLMs) on their ability to use tools under complex constraints. It includes 200 test cases that average seven constraint types and 4,700‑token prompts, covering resource, behavior, toolset, and response dimensions. An executable validation module performs step‑level checks, and nine state‑of‑the‑art LLMs were tested, revealing that none exceed a 20% task completion rate when strict constraints are enforced, with frequent violations and limited self‑refinement.

By Junjie Ye, Guoqiang Zhang, Wenjie Fu, Zelin Li, Tao Gui, Qi Zhang, Xuanjing Huang
arXiv Computation and Language
Oct 15

BANGLAPLAN: A Benchmark for Constraint-Based Planning and Decision-Making in Bangla

BANGLAPLAN is a new benchmark comprising 1,000 Bangla planning and decision‑making problems across ten everyday task families, presented in four language registers (everyday, formal, colloquial, and Bangla–English mixed). The benchmark includes both feasible and infeasible instances and requires answers in eight structured JSON formats such as schedules, routes, and allocations. Evaluation of 11 language‑model configurations shows a wide performance gap, with GPT‑6.1 Sol achieving 98.7% accuracy while most models perform poorly, highlighting the challenge of constraint‑based planning in Bangla.

By Md Rayhanul Masud, S M Maksudul Alam, Mohammad Jaminur Islam