The paper evaluates the planning abilities of large language model (LLM) agents by comparing their performance against formal semantics defined in hidden PDDL models. Across five domains, four LLMs, six prompting strategies, and three agent architectures, the best setup solved 78% of solvable tasks offline but only 40% online, with limited recognition of unsolvable tasks and no clear benefit from more elaborate prompting. The study concludes that claims of LLM planning competence require validated action traces rather than end‑to‑end success rates.
By Oriol Miro-Lopez-Feliu, Oscar Mir\'o L\'opez-Feliu, Victor Gimenez-Abalos, Javier Vazquez-Salceda, Victor Carballo Araruna, Sergio Alvarez-Napagao
arXiv:2608.24888v2 Announce Type: replace
Abstract: Agents that act on a user's behalf must plan differently for different users, and increasingly do so from some structured representation of user co...
By Chirag Shah
The paper introduces the ReSHAPE task, which evaluates how planning agents can revise user specifications to better match acceptable plans by reshaping the search space. Using the ReSHAPE Planter framework on travel and shopping benchmarks, the authors analyze 5,800 agent-revised specifications and find that a single correct trade‑off can increase the planner’s success rate by up to 4.6×. Current agents admit only 32% of acceptable plans, whereas incorporating trade‑offs raises acceptance to over 52%, and showing concrete plans improves F1 scores by 9.7 points compared to merely asking about preferences.
By Yilin Zhang, Xinran Zhao, Tong Chen, Ziming Yuan, Tongshuang Wu
The paper argues that evaluating agents as fixed models is insufficient, proposing instead to treat them as configurable systems. Using a new benchmark of four scientific tasks, the authors analyze how five configuration aspects—task information, reasoning, self‑verification, time budget, and backbone model—affect performance, noting that about 54% of outcome variance arises from run‑to‑run differences even with the same settings. The study finds that providing more task information has the strongest impact, while interactions among settings (e.g., extra time only helps with adequate information or model capability) and the choice of verification tools significantly shape agent behavior.
By Luis Wiedmann, Leander Girrbach, Cordelia Schmid, Zeynep Akata
arXiv:2610.41129v1 Announce Type: new
Abstract: AI agents increasingly carry out delegated tasks in parallel, with one person overseeing several of them at once. In practice, an agent often lacks inf...
By Hanzhao Wang, Shengjie Liu, Zhengchao Wang
The paper introduces When2Tool, a benchmark of 18 environments that categorizes tasks by whether a tool call is necessary, covering computational scale, knowledge boundaries, and execution reliability. It evaluates two training‑free baselines—Prompt‑only and Reason‑then‑Act—finding they inadequately control tool usage. By probing hidden states, the authors discover that models can predict tool necessity with high accuracy, leading to the Probe&Prefill method that reduces unnecessary tool calls by 38% with minimal accuracy loss.
By Chung-En Sun, Linbo Liu, Ge Yan, Zimo Wang, Tsui-Wei Weng
arXiv:2610.17757v1 Announce Type: new
Abstract: Large language model agents can follow complex human instructions to solve real-world tasks. In practice, however, initial instructions are often incom...
By Kaiming Liu, Yuxuan Liu, Jinrui Ju, Minyuan Ruan, Xuanyu Lei, Peng Li, Yang Liu
arXiv:2608. 14808v1 Announce Type: new Abstract: When a user question is underspecified, a capable model should recognize that its context is insufficient, identify the missing information, ask for it, and respond only once that information determines a unique answer.
By Yepeng Huang, Jiawen Zhang, Michelle Dai, Xiaorui Su, Shanghua Gao, Zi Wang, Marinka Zitnik
The paper introduces the ReSHAPE task, which evaluates how planning agents can revise user specifications to better match acceptable plans by reshaping the search space. Using the ReSHAPE Planter framework, the authors analyze 5,800 agent-revised specifications in travel and shopping scenarios, showing that a single correct trade‑off can increase the likelihood of finding an acceptable plan by up to 4.6× and that search space quality predicts planner success. Current agents admit only 32% of acceptable plans, whereas incorporating trade‑offs raises acceptance to over 52%, and showing concrete plans improves F1 scores by 9.7 points compared to merely asking preferences.
whyItMatters":"The study demonstrates that allowing agents to reshape user requests and present concrete plans significantly enhances their ability to find acceptable solutions, highlighting a practical improvement for planning systems."
arXiv:2609.37588v1 Announce Type: new
Abstract: Users of language-based assistive agents often make ambiguous requests. In response, an assistant can either directly act on its interpretation of the...
By T. Duy Nguyen-Hien, Yee Whye Teh, Wee Sun Lee, Tan Zhi-Xuan
Users of language-based assistive agents often make ambiguous requests. In response, an assistant can either directly act on its interpretation of the request --- risking misalignment with the user --...
arXiv:2609.37236v1 Announce Type: new
Abstract: An agent that uses tools typically responds to what the user explicitly asks, yet completing the task may require information the user never requested....
By Ido Levy, Asaf Yehudai, Segev Shlomov, Asaf Adi, Leshem Choshen