arXiv AI

How Do Agent Harnesses Create Value? Planning Information and Release Control in Stateful LLM Agents

The paper investigates how agent harnesses—specifically planning guidance, execution organization, and completion verification—affect performance in retail and airline pilot tasks. By comparing fixed, task‑specific plans to shuffled policy text of equal length, the study finds that fixed plans improve success rates by about 7 percentage points, especially on complex tasks. A read‑only verifier rejects a majority of invalid episodes while incurring minimal cost, and its impact varies with the penalty for erroneous acceptance, often matching the full planning‑plus‑verification benefit at a lower cost.

arXiv AI
6d ago

From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings

The paper introduces Ask2Decide, a benchmark of 43 planning tasks designed to test whether large language model (LLM) agents can identify and request missing decision‑relevant information from humans. It collects 79 interaction sessions from 19 participants, showing that human replies often omit critical facts, and uses these data to calibrate a simulated operator. Evaluation of four state‑of‑the‑art LLMs reveals that their initial plans are infeasible on 58‑98% of tasks when information is missing, and that agents only slightly outperform random guessing in distinguishing useful from irrelevant facts.

By Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui, Ao Qu, Ou Zheng, Jiangbo Yu, Haris N. Koutsopoulos, Jinhua Zhao, Yuan Zhou
arXiv Machine Learning
Oct 19

Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers

The paper introduces SPLIT, a method that optimally allocates test‑time compute among sampling, model selection, and verification for language models and agents. Using an exact probability formula that accounts for task difficulty, shared blind spots, and verifier correlations, SPLIT is fitted via empirical Bayes on a small labeled pilot and then used to plan budgets for unseen tasks. Across seven diverse benchmarks, 26 models, and six verifier types, SPLIT achieves near‑optimal performance, outperforming rule‑based baselines by up to 2.5 percentage points and providing insights into when additional LLM judges reduce precision.

By Srinivasan Iyengar
arXiv AI
6d ago

Veritas: A Verifier Generation Harness for Agent Tasks

Veritas is a harness that automatically generates verifiers for agent tasks, adding an evidence header, a library of domain rules, and authoring principles refined through a self‑improvement loop. It was evaluated on six benchmarks, outperforming seven other harnesses on all four execution‑scored tasks and matching or exceeding performance on pairwise comparison tasks. The system also improves agreement with official verdicts across multiple model families and harness‑benchmark pairs.

By Daoan Zhang, Qinbo Bai, Dehai Min, Xingyu Mu, Ziyi Chen, Jianing Yin
arXiv AI
Nov 2

Evidence-Guided Recovery for AI Agents with Ambiguous Tool Outcomes and Partial Execution

The paper introduces an evidence‑guided recovery framework for AI agents that encounter ambiguous tool outcomes and partial execution, such as timeouts or failures. It defines a shared admissibility check for key expiry, authorization timing, dependencies, and residual effects, and a terminal certificate to distinguish actual goal satisfaction from publicly supported safe completion. In a finite study of 37 assignments, the approach achieved 28 certified completions, and in a larger comparison of 58 paired units, evidence‑guided advice certified 30 completions versus 9 for evidence prompting, demonstrating a significant improvement in task completion rates.

By Genliang Zhu, Xiaobo Chen, Ziyuan Wang, Chu Wang
arXiv AI
Oct 19

Testing the Planning Claims of LLM-Based Agents against Formal Semantics

The paper evaluates the planning abilities of large language model (LLM) agents by comparing their performance against formal semantics defined in hidden PDDL models. Across five domains, four LLMs, six prompting strategies, and three agent architectures, the best setup solved 78% of solvable tasks offline but only 40% online, with limited recognition of unsolvable tasks and no clear benefit from more elaborate prompting. The study concludes that claims of LLM planning competence require validated action traces rather than end‑to‑end success rates.

By Oriol Miro-Lopez-Feliu, Oscar Mir\'o L\'opez-Feliu, Victor Gimenez-Abalos, Javier Vazquez-Salceda, Victor Carballo Araruna, Sergio Alvarez-Napagao
arXiv AI
Sep 30

The Default Trap: Rethinking Plan Evaluation in Tool-Using LLM Agents

The paper investigates how large language model agents that use tools respond to changes in plan priorities versus default plan removal, a phenomenon termed the "default trap." Experiments across 3,200 decision windows on Retail, Airline, and AgentDojo tasks show that switching priorities strongly redirects model choices, while removing a default plan yields weaker responsiveness. Additional studies reveal that the order of account lists and the presence of extra text significantly influence default target selection and priority effects, with overall task success varying from -19.4 to +8.3 points relative to no plan.

By Xueqi Li, Jingjie Ning, Yibo Kong
arXiv AI
Oct 21

When Does an Added Planning Capability Create Operational Value? An Observable Evidence Standard

The paper introduces an observable evidence standard to assess whether adding a planning capability in industrial AI systems delivers operational value. By separating opportunity, plan change, consequence, equivalence, uncertainty, and scope, the authors evaluate diverse planning additions on real retail datasets, revealing that composite policies can mask negligible contributions, execution changes may not always yield benefits, and indirect evidence alone cannot fully attribute value. The study demonstrates how this evidence record can guide decisions on validation, deployment, monitoring, redesign, and retirement of planning capabilities.

By Yihua Xu, Yan Lyu, Jize Li, Jiani He, You Lin, Yidan Hu, Rubing Qiang