arXiv AI By Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui, Ao Qu, Ou Zheng, Jiangbo Yu, Haris N. Koutsopoulos, Jinhua Zhao, Yuan Zhou

From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings

Read the original on arXiv AI →

The paper introduces Ask2Decide, a benchmark of 43 planning tasks designed to test whether large language model (LLM) agents can identify and request missing decision‑relevant information from humans. It collects 79 interaction sessions from 19 participants, showing that human replies often omit critical facts, and uses these data to calibrate a simulated operator. Evaluation of four state‑of‑the‑art LLMs reveals that their initial plans are infeasible on 58‑98% of tasks when information is missing, and that agents only slightly outperform random guessing in distinguishing useful from irrelevant facts.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Oct 19

Testing the Planning Claims of LLM-Based Agents against Formal Semantics

The paper evaluates the planning abilities of large language model (LLM) agents by comparing their performance against formal semantics defined in hidden PDDL models. Across five domains, four LLMs, six prompting strategies, and three agent architectures, the best setup solved 78% of solvable tasks offline but only 40% online, with limited recognition of unsolvable tasks and no clear benefit from more elaborate prompting. The study concludes that claims of LLM planning competence require validated action traces rather than end‑to‑end success rates.

By Oriol Miro-Lopez-Feliu, Oscar Mir\'o L\'opez-Feliu, Victor Gimenez-Abalos, Javier Vazquez-Salceda, Victor Carballo Araruna, Sergio Alvarez-Napagao
arXiv AI
3d ago

How Should Planning Agents ReSHAPE Their Search Space?

The paper introduces the ReSHAPE task, which evaluates how planning agents can revise user specifications to better match acceptable plans by reshaping the search space. Using the ReSHAPE Planter framework on travel and shopping benchmarks, the authors analyze 5,800 agent-revised specifications and find that a single correct trade‑off can increase the planner’s success rate by up to 4.6×. Current agents admit only 32% of acceptable plans, whereas incorporating trade‑offs raises acceptance to over 52%, and showing concrete plans improves F1 scores by 9.7 points compared to merely asking about preferences.

By Yilin Zhang, Xinran Zhao, Tong Chen, Ziming Yuan, Tongshuang Wu
arXiv AI
Oct 2

Agents Are Systems, Not Models: Rethinking Agentic Evaluation

The paper argues that evaluating agents as fixed models is insufficient, proposing instead to treat them as configurable systems. Using a new benchmark of four scientific tasks, the authors analyze how five configuration aspects—task information, reasoning, self‑verification, time budget, and backbone model—affect performance, noting that about 54% of outcome variance arises from run‑to‑run differences even with the same settings. The study finds that providing more task information has the strongest impact, while interactions among settings (e.g., extra time only helps with adequate information or model capability) and the choice of verification tools significantly shape agent behavior.

By Luis Wiedmann, Leander Girrbach, Cordelia Schmid, Zeynep Akata
arXiv Computation and Language
Nov 5

LLM Agents Already Know When to Call Tools -- Even Without Reasoning

The paper introduces When2Tool, a benchmark of 18 environments that categorizes tasks by whether a tool call is necessary, covering computational scale, knowledge boundaries, and execution reliability. It evaluates two training‑free baselines—Prompt‑only and Reason‑then‑Act—finding they inadequately control tool usage. By probing hidden states, the authors discover that models can predict tool necessity with high accuracy, leading to the Probe&Prefill method that reduces unnecessary tool calls by 38% with minimal accuracy loss.

By Chung-En Sun, Linbo Liu, Ge Yan, Zimo Wang, Tsui-Wei Weng