The paper investigates how different natural language plan representations affect the performance of large language model (LLM) web agents. Using the PlanAhead framework, the authors automatically classify WebArena tasks into three difficulty levels and evaluate four plan types—sequential subgoals, narrative, pseudocode, and checklist—on hard tasks across multiple multimodal LLM agents. They introduce a decomposition of Success Rate into Achievement Rate and Solved-Task Consistency to better understand stochastic variability, finding that both plan formulation and the LLM used to generate the plan significantly influence agent robustness and success.
arXiv:2508.01300v2 Announce Type: replace Abstract: AI planning provides agents the ability to act autonomously towards achieving a goal in a given environment. Recent advances in Large Language Mode...
arXiv:2607. 05775v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly evaluated on their ability to use tools, plan multi-step tasks, coordinate with other agents, and operate over extended horizons.
arXiv:2606. 27330v1 Announce Type: cross Abstract: Multimodal web agents can assist humans in operating repetitive GUI tasks, where effective task planning is essential for decomposing complex tasks into executable actions.
Large language model (LLM) agents are increasingly evaluated on their ability to use tools, plan multi-step tasks, coordinate with other agents, and operate over extended horizons. Reported benchmark gains often obscure recurring failure modes documented across otherwise unrelated evaluation efforts.
Agentick is a unified benchmark for sequential decision‑making agents that evaluates RL, LLM, VLM, hybrid, and human agents on 37 procedurally generated tasks across six capability categories, four difficulty levels, and five observation modalities via a single Gymnasium‑compatible interface. It includes a Coding API, oracle reference policies, pre‑built SFT datasets, a composable agent harness, and a live leaderboard. An evaluation of 27 configurations and over 90,000 episodes shows no single approach dominates, with GPT‑5 mini leading overall, PPO excelling in planning and multi‑agent tasks, and the reasoning harness boosting LLM performance by 3‑10×, while ASCII observations outperform natural language.
The paper evaluates the planning abilities of large language model (LLM) agents by comparing their performance against formal semantics defined in hidden PDDL models. Across five domains, four LLMs, six prompting strategies, and three agent architectures, the best setup solved 78% of solvable tasks offline but only 40% online, with limited recognition of unsolvable tasks and no clear benefit from more elaborate prompting. The study concludes that claims of LLM planning competence require validated action traces rather than end‑to‑end success rates.
M-Plan is a closed‑loop multi‑agent system that combines long‑term and short‑term memory to improve large language model agents on long‑horizon, multi‑turn tasks. It uses task‑specific milestones and local hints to generate global‑local guidance, forming short‑term memory during execution, and then optimizes and stores relevant information in long‑term memory for future use. Experiments on ALFWorld, Webshop, and PDDL show that M‑Plan outperforms baseline methods in both accuracy and efficiency.
IDRBench is a benchmark designed to evaluate the interactive capabilities of deep research agents that use large language models. It introduces controlled opportunities for clarification within a common workflow, comparing autonomous and interactive trajectories by measuring task‑specific report alignment and interaction cost. Experiments on 100 tasks with seven LLMs show that interaction consistently improves alignment, though its effectiveness varies depending on the agents’ questions and feedback integration.
arXiv:2604. 00594v2 Announce Type: replace Abstract: As the focus in LLM-based coding shifts from static single-step code generation to multi-step agentic interaction with tools and environments, understanding which tasks will challenge agents and why becomes increasingly difficult.
The paper investigates how to improve large language model (LLM) based autonomous web agents by filtering irrelevant webpage content. The authors reproduce baseline models on the WebArena benchmark and identify failure modes caused by raw HTML input. They propose DeBERTa‑ and T5‑based retrieval models that rank HTML elements by relevance, fine‑tuned on Mind2Web data, and demonstrate that the DeBERTa model raises the LLaMA‑2‑70B agent’s success rate from 1.97% to 2.96%. Additionally, a zero‑shot ColBERT retriever achieves recall of 0.52 on Mind2Web and 0.47 on WebArena.
PlanPO introduces a group planning-aware policy optimization method for multi-turn agentic large language models, addressing the issue of advantage collapse caused by treating all successful trajectories equally. By incorporating coarse-to-fine advantage signals that reflect differences in trajectory and turn lengths, PlanPO encourages agents to learn generalizable planning and generation behaviors. Experiments show a 27.2% average improvement over GRPO on benchmarks such as ALFWorld, WebShop, and SciWorld, with minimal extra training cost.
arXiv:2608. 02650v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly rely on external tools to complete complex real-world tasks.