arXiv AI By Yilin Zhang, Xinran Zhao, Tong Chen, Ziming Yuan, Tongshuang Wu

How Should Planning Agents ReSHAPE Their Search Space?

Read the original on arXiv AI →

The paper introduces the ReSHAPE task, which evaluates how planning agents can revise user specifications to better match acceptable plans by reshaping the search space. Using the ReSHAPE Planter framework on travel and shopping benchmarks, the authors analyze 5,800 agent-revised specifications and find that a single correct trade‑off can increase the planner’s success rate by up to 4.6×. Current agents admit only 32% of acceptable plans, whereas incorporating trade‑offs raises acceptance to over 52%, and showing concrete plans improves F1 scores by 9.7 points compared to merely asking about preferences.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Hugging Face Trending Papers
4d ago

How Should Planning Agents ReSHAPE Their Search Space?

The paper introduces the ReSHAPE task, which evaluates how planning agents can revise user specifications to better match acceptable plans by reshaping the search space. Using the ReSHAPE Planter framework, the authors analyze 5,800 agent-revised specifications in travel and shopping scenarios, showing that a single correct trade‑off can increase the likelihood of finding an acceptable plan by up to 4.6× and that search space quality predicts planner success. Current agents admit only 32% of acceptable plans, whereas incorporating trade‑offs raises acceptance to over 52%, and showing concrete plans improves F1 scores by 9.7 points compared to merely asking preferences. whyItMatters":"The study demonstrates that allowing agents to reshape user requests and present concrete plans significantly enhances their ability to find acceptable solutions, highlighting a practical improvement for planning systems."

arXiv AI
6d ago

From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings

The paper introduces Ask2Decide, a benchmark of 43 planning tasks designed to test whether large language model (LLM) agents can identify and request missing decision‑relevant information from humans. It collects 79 interaction sessions from 19 participants, showing that human replies often omit critical facts, and uses these data to calibrate a simulated operator. Evaluation of four state‑of‑the‑art LLMs reveals that their initial plans are infeasible on 58‑98% of tasks when information is missing, and that agents only slightly outperform random guessing in distinguishing useful from irrelevant facts.

By Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui, Ao Qu, Ou Zheng, Jiangbo Yu, Haris N. Koutsopoulos, Jinhua Zhao, Yuan Zhou
arXiv AI
Oct 19

Testing the Planning Claims of LLM-Based Agents against Formal Semantics

The paper evaluates the planning abilities of large language model (LLM) agents by comparing their performance against formal semantics defined in hidden PDDL models. Across five domains, four LLMs, six prompting strategies, and three agent architectures, the best setup solved 78% of solvable tasks offline but only 40% online, with limited recognition of unsolvable tasks and no clear benefit from more elaborate prompting. The study concludes that claims of LLM planning competence require validated action traces rather than end‑to‑end success rates.

By Oriol Miro-Lopez-Feliu, Oscar Mir\'o L\'opez-Feliu, Victor Gimenez-Abalos, Javier Vazquez-Salceda, Victor Carballo Araruna, Sergio Alvarez-Napagao
arXiv AI
Oct 2

Agents Are Systems, Not Models: Rethinking Agentic Evaluation

The paper argues that evaluating agents as fixed models is insufficient, proposing instead to treat them as configurable systems. Using a new benchmark of four scientific tasks, the authors analyze how five configuration aspects—task information, reasoning, self‑verification, time budget, and backbone model—affect performance, noting that about 54% of outcome variance arises from run‑to‑run differences even with the same settings. The study finds that providing more task information has the strongest impact, while interactions among settings (e.g., extra time only helps with adequate information or model capability) and the choice of verification tools significantly shape agent behavior.

By Luis Wiedmann, Leander Girrbach, Cordelia Schmid, Zeynep Akata
arXiv AI
Aug 28

Behavior2Trip: Towards Personalized Travel Planning via User Behavior Trajectory

Behavior2Trip introduces a new task—Behavior‑Aware Travel Planning—where user preferences are inferred from past behavior trajectories rather than explicit instructions. The benchmark contains 11,400 instances from a major Chinese travel platform, each with nearly 40 recorded behaviors across 14 attributes and 5 preference dimensions. A reinforcement‑learning agent, B2T‑Agent, leverages these trajectories, external retrieval tools, and internal memory, outperforming GPT‑4.1 and other baselines on the dataset.

By Zihao Cheng, Yingyu Shan, Hongru Wang, Zeming Liu, Xinyi Wang, Xiangrong Zhu, Yuhang Guo, Wei Lin, Yunhong Wang