arXiv AI
Aug 7
EcoAgent-Bench: Evaluating Economic Decision-Making in Budget-Constrained LLM Agents
arXiv:2608. 05519v1 Announce Type: new Abstract: Agent benchmarks usually measure task completion and treat resource use as an auxiliary statistic.
$R^3$-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets
Related stories
arXiv Machine Learning
Oct 3
How Much Can Language Models Gain from Test-Time Computation?
The paper investigates how test‑time computation can enhance language models and at what cost, introducing the SELF‑POT benchmark to evaluate this across competition mathematics, competitive programming, and agentic workflows. SELF‑POT separates candidate coverage from final accuracy, tracks correctness transitions under revision, and measures protocol completion alongside task success. Using a unified budget rule, the study compares direct inference, parallel sampling, and self‑revision across five low‑cost reasoning models, revealing that selection rules and failure handling significantly influence gains and cost savings.
arXiv AI
Jun 4
Not All Errors Are Equal: Consequence-Aware Reasoning Compute Allocation
arXiv:2606. 04402v1 Announce Type: new Abstract: Modern reasoning models can allocate different amounts of test-time computation, such as thinking tokens, model calls, or compute budget, to different tasks.
arXiv AI
Sep 30
Thinking Before Thinking: Scaling Agentic Inference Through Meta-Reasoning
The paper introduces agentic meta‑reasoning, a structured inference‑time framework that explicitly manages control decisions—such as selecting partial work, restarting, or stopping—during long‑horizon agentic tasks. By delegating task execution to workers and consolidating decisions through a lightweight controller that references persistent memory, the method reduces the need to replay full histories. Experiments on ProgramBench and other benchmarks show that meta‑reasoning improves performance over direct control baselines, especially as computation budgets increase, and reveals greater reuse of earlier work and higher solution coverage.
arXiv AI
Oct 27
What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents
The paper investigates how users should set the reasoning‑effort level in large language models, a parameter that trades off inference cost for potential performance gains. Experiments on proprietary and open‑weight models across four benchmarks show that higher effort does not consistently improve accuracy, can be up to 42× more expensive on easy tasks, and alters agent behavior by increasing verification and search steps. The authors propose a simple escalation strategy—starting at the lowest effort and only increasing after a verified failure—that matches or outperforms fixed high‑effort settings while reducing cost.
arXiv AI
Oct 21
PORA: Learning When More Reasoning Pays Off for LLM Agents
The paper introduces PORA, a method that teaches large language model agents to decide when to allocate more reasoning steps versus fewer, based on the expected downstream benefit. PORA trains by running each state twice—once with short reasoning and once with long reasoning—and learns to predict the value of extended reasoning. During inference, a controller uses this prediction and remaining budget information to choose the appropriate reasoning length without running both options, improving performance across diverse tasks.
Hugging Face Trending Papers
Aug 2
Same Task, Different Work: Prompt-Induced Waste in Coding Agents
Two prompts can request the same code change and produce the same correct patch, yet cause a coding agent to perform radically different kinds and amounts of work. We study this effect in a preregistered benchmark spanning 4,644 valid runs, 24 deterministic coding tasks, seven reasoning models, and two real agent harnesses.
arXiv AI
Jun 30
CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents
arXiv:2511. 02734v3 Announce Type: replace Abstract: Current evaluations of Large Language Model (LLM) agents primarily emphasize task completion, often overlooking resource efficiency and adaptability.
arXiv AI
Jul 15
Do AI Agents Know When a Task Is Simple? Toward Complexity-Aware Reasoning and Execution
arXiv:2607. 13034v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly automate multi-step engineering and informatics workflows, yet they rarely ask how much effort a task actually requires.
arXiv AI
Aug 28
Don't Overthink, Don't Underthink: Toward Adaptive Reasoning in Agentic AI
The paper argues that large language models need adaptive reasoning rather than fixed reasoning budgets. It shows that over‑reasoning leads to high computational cost without accuracy gains, while under‑reasoning results in incorrect or incomplete solutions. The authors evaluate these failure modes on MATH‑500 and the GAIA benchmark, highlighting the need for dynamic reasoning allocation in agentic AI systems.
Hugging Face Trending Papers
Oct 19
BOSS: Token-Budgeted Online Sampling and Selection for LLM Reasoning
Test-time scaling can improve LLM reasoning, but existing approaches typically adapt generation or apply answer selection after a predetermined generation schedule. We study within-instance allocation...
Hugging Face Trending Papers
Sep 24
GRASP: Generating, Revising, and Assessing for Strategic Planning with Agentic AI
GRASP is a multi-stage planning framework that separates planning into specialized modules: GenPlan for global macro-guidelines, RevPlan for exploring localized strategies, and VerPlan for multi-criteria evaluation. This strategy-aware approach yields state‑of‑the‑art accuracy on diverse datasets, outperforming direct LLM planners by up to 30.8% on ZebraLogic and reducing multi‑task degradation. GRASP’s context isolation and macro‑regularization also give it a 14.5% edge over frontier reasoning models like GPT‑5‑mini.