EcoAgent-Bench: Evaluating Economic Decision-Making in Budget-Constrained LLM Agents
arXiv:2608. 05519v1 Announce Type: new Abstract: Agent benchmarks usually measure task completion and treat resource use as an auxiliary statistic.
The paper investigates how agent harnesses—specifically planning guidance, execution organization, and completion verification—affect performance in retail and airline pilot tasks. By comparing fixed, task‑specific plans to shuffled policy text of equal length, the study finds that fixed plans improve success rates by about 7 percentage points, especially on complex tasks. A read‑only verifier rejects a majority of invalid episodes while incurring minimal cost, and its impact varies with the penalty for erroneous acceptance, often matching the full planning‑plus‑verification benefit at a lower cost.
arXiv:2608. 05519v1 Announce Type: new Abstract: Agent benchmarks usually measure task completion and treat resource use as an auxiliary statistic.
arXiv:2604. 12147v3 Announce Type: replace-cross Abstract: Agents are commonly instructed to follow a task-specific plan for guidance.
The paper introduces Ask2Decide, a benchmark of 43 planning tasks designed to test whether large language model (LLM) agents can identify and request missing decision‑relevant information from humans. It collects 79 interaction sessions from 19 participants, showing that human replies often omit critical facts, and uses these data to calibrate a simulated operator. Evaluation of four state‑of‑the‑art LLMs reveals that their initial plans are infeasible on 58‑98% of tasks when information is missing, and that agents only slightly outperform random guessing in distinguishing useful from irrelevant facts.
The paper introduces SPLIT, a method that optimally allocates test‑time compute among sampling, model selection, and verification for language models and agents. Using an exact probability formula that accounts for task difficulty, shared blind spots, and verifier correlations, SPLIT is fitted via empirical Bayes on a small labeled pilot and then used to plan budgets for unseen tasks. Across seven diverse benchmarks, 26 models, and six verifier types, SPLIT achieves near‑optimal performance, outperforming rule‑based baselines by up to 2.5 percentage points and providing insights into when additional LLM judges reduce precision.
arXiv:2610.16491v1 Announce Type: cross Abstract: Improving an agent through successive revisions requires balancing exploration and evaluation under a limited budget. A narrow search can miss useful...
Veritas is a harness that automatically generates verifiers for agent tasks, adding an evidence header, a library of domain rules, and authoring principles refined through a self‑improvement loop. It was evaluated on six benchmarks, outperforming seven other harnesses on all four execution‑scored tasks and matching or exceeding performance on pairwise comparison tasks. The system also improves agreement with official verdicts across multiple model families and harness‑benchmark pairs.
The paper introduces an evidence‑guided recovery framework for AI agents that encounter ambiguous tool outcomes and partial execution, such as timeouts or failures. It defines a shared admissibility check for key expiry, authorization timing, dependencies, and residual effects, and a terminal certificate to distinguish actual goal satisfaction from publicly supported safe completion. In a finite study of 37 assignments, the approach achieved 28 certified completions, and in a larger comparison of 58 paired units, evidence‑guided advice certified 30 completions versus 9 for evidence prompting, demonstrating a significant improvement in task completion rates.
Industrial artificial intelligence (AI) planning systems combine predictive models, optimization logic, controls, and governance rules, but improvement in the complete policy does not show whether one...
The paper evaluates the planning abilities of large language model (LLM) agents by comparing their performance against formal semantics defined in hidden PDDL models. Across five domains, four LLMs, six prompting strategies, and three agent architectures, the best setup solved 78% of solvable tasks offline but only 40% online, with limited recognition of unsolvable tasks and no clear benefit from more elaborate prompting. The study concludes that claims of LLM planning competence require validated action traces rather than end‑to‑end success rates.
arXiv:2610.18614v1 Announce Type: new Abstract: Controlling an acting LLM agent means deciding, at each step of a long-horizon task, whether to trust its next action, abstain, or hand off to a human....
The paper investigates how large language model agents that use tools respond to changes in plan priorities versus default plan removal, a phenomenon termed the "default trap." Experiments across 3,200 decision windows on Retail, Airline, and AgentDojo tasks show that switching priorities strongly redirects model choices, while removing a default plan yields weaker responsiveness. Additional studies reveal that the order of account lists and the presence of extra text significantly influence default target selection and priority effects, with overall task success varying from -19.4 to +8.3 points relative to no plan.
The paper introduces an observable evidence standard to assess whether adding a planning capability in industrial AI systems delivers operational value. By separating opportunity, plan change, consequence, equivalence, uncertainty, and scope, the authors evaluate diverse planning additions on real retail datasets, revealing that composite policies can mask negligible contributions, execution changes may not always yield benefits, and indirect evidence alone cannot fully attribute value. The study demonstrates how this evidence record can guide decisions on validation, deployment, monitoring, redesign, and retirement of planning capabilities.