The paper evaluates the planning abilities of large language model (LLM) agents by comparing their performance against formal semantics defined in hidden PDDL models. Across five domains, four LLMs, six prompting strategies, and three agent architectures, the best setup solved 78% of solvable tasks offline but only 40% online, with limited recognition of unsolvable tasks and no clear benefit from more elaborate prompting. The study concludes that claims of LLM planning competence require validated action traces rather than end‑to‑end success rates.
By Oriol Miro-Lopez-Feliu, Oscar Mir\'o L\'opez-Feliu, Victor Gimenez-Abalos, Javier Vazquez-Salceda, Victor Carballo Araruna, Sergio Alvarez-Napagao
arXiv:2608.24888v2 Announce Type: replace
Abstract: Agents that act on a user's behalf must plan differently for different users, and increasingly do so from some structured representation of user co...
By Chirag Shah
The paper introduces the ReSHAPE task, which evaluates how planning agents can revise user specifications to better match acceptable plans by reshaping the search space. Using the ReSHAPE Planter framework on travel and shopping benchmarks, the authors analyze 5,800 agent-revised specifications and find that a single correct trade‑off can increase the planner’s success rate by up to 4.6×. Current agents admit only 32% of acceptable plans, whereas incorporating trade‑offs raises acceptance to over 52%, and showing concrete plans improves F1 scores by 9.7 points compared to merely asking about preferences.
By Yilin Zhang, Xinran Zhao, Tong Chen, Ziming Yuan, Tongshuang Wu
The paper argues that evaluating agents as fixed models is insufficient, proposing instead to treat them as configurable systems. Using a new benchmark of four scientific tasks, the authors analyze how five configuration aspects—task information, reasoning, self‑verification, time budget, and backbone model—affect performance, noting that about 54% of outcome variance arises from run‑to‑run differences even with the same settings. The study finds that providing more task information has the strongest impact, while interactions among settings (e.g., extra time only helps with adequate information or model capability) and the choice of verification tools significantly shape agent behavior.
By Luis Wiedmann, Leander Girrbach, Cordelia Schmid, Zeynep Akata
arXiv:2610.41129v1 Announce Type: new
Abstract: AI agents increasingly carry out delegated tasks in parallel, with one person overseeing several of them at once. In practice, an agent often lacks inf...
By Hanzhao Wang, Shengjie Liu, Zhengchao Wang
The paper introduces When2Tool, a benchmark of 18 environments that categorizes tasks by whether a tool call is necessary, covering computational scale, knowledge boundaries, and execution reliability. It evaluates two training‑free baselines—Prompt‑only and Reason‑then‑Act—finding they inadequately control tool usage. By probing hidden states, the authors discover that models can predict tool necessity with high accuracy, leading to the Probe&Prefill method that reduces unnecessary tool calls by 38% with minimal accuracy loss.
By Chung-En Sun, Linbo Liu, Ge Yan, Zimo Wang, Tsui-Wei Weng