arXiv AI

The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

The paper introduces the concept of an agent’s "taste"—its ability to make effective long‑horizon decisions—and presents Taste‑Bench, a new benchmark that automatically generates decision‑fork questions from agent trajectories. Taste‑Bench evaluates models on choosing the best path without seeing future outcomes, revealing that top models answer only about 60% of questions correctly and that later‑appearing evidence makes forks harder. The authors also demonstrate that training a student model to mimic a teacher’s judgment improves decision quality and overall success on held‑out software engineering tasks.

arXiv AI
Oct 19

EvalResearchBench: Can AI Agents Build Their Own Feedback for Recursive Self-Improvement?

EvalResearchBench (ERB) is a benchmark that tests whether AI agents can autonomously design evaluation tasks for recursive self‑improvement. In ERB, a researcher agent selects or synthesizes tasks, builds graders, and refines them through pilot tests before producing a final evaluator that runs within fixed time and API budgets. Experiments with nine researchers and thirteen candidate models show that the best automated evaluators achieve about 75% concordance with target benchmarks, below the 91% ceiling set by target disagreements, and that human‑designed task samples still outperform many automated approaches.

By Yaolun Zhang, Tianyi Xu, Yujie Zhao, Jishen Zhao, Qingyun Wu, Huazheng Wang
arXiv AI
Oct 27

What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

The paper investigates how users should set the reasoning‑effort level in large language models, a parameter that trades off inference cost for potential performance gains. Experiments on proprietary and open‑weight models across four benchmarks show that higher effort does not consistently improve accuracy, can be up to 42× more expensive on easy tasks, and alters agent behavior by increasing verification and search steps. The authors propose a simple escalation strategy—starting at the lowest effort and only increasing after a verified failure—that matches or outperforms fixed high‑effort settings while reducing cost.

By Kaiser Sun, Yongchan Kwon, Zain Hasan, Shang Zhu, Kaitlyn Zhou, Federico Bianchi
arXiv AI
3d ago

Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verification

The paper introduces SGV, a self‑grounded verification method that mitigates agreement bias in multimodal large language models (MLLMs). By first generating broad priors about desired behavior and then conditioning on those priors to evaluate agent trajectories, SGV improves human alignment and detection of failures across web navigation, computer use, and robotics tasks. Experiments on 13 models and 28 judge designs show up to 25 percentage‑point gains in failure detection and significant performance boosts in downstream applications such as GUI task completion and robotic policy learning.

By Moises Andrade, Joonhyuk Cha, Brandon Ho, Vriksha Srihari, Karmesh Yadav, Zsolt Kira
arXiv AI
Oct 21

GRADUS: Discovering and Recovering Agent Lessons from the Environment's Latent Syllabus

GRADUS introduces a method for training large language model agents in policy-constrained tool environments by decoupling task design from solutions. It treats a fixed environment as a latent syllabus, extracting guarded behavioral operators and composing them into progressively harder tasks while using corpus-level feedback to target missing outcome combinations. The approach jointly calibrates tasks and solutions against the environment, turning verification into a recovery signal that preserves difficult examples, leading to significant performance gains on benchmarks such as $ au^2$-Bench and CAR-Bench.

By Qing Ping, Wei Yang, Panpan Xu, Johannes Kirmayr, Lukas Stappen, Junjie Tang, Jun Huan
arXiv AI
Oct 27

The Hitchhiker's Guide to System 1 Models: Fast Decision-Making Beyond LLMS

The paper introduces System 1 models—fast, single‑pass decision engines that map inputs directly to structured outputs, addressing the latency, cost, and calibration issues of large language models. It presents a five‑family taxonomy (encoder‑based classifiers, reward models, non‑autoregressive engines, contrastive models, and extraction models) and discusses training, deployment trade‑offs, and calibration theory, including proper scoring rules and RLCD. The guide also covers integration into agentic workflows, System 1+System 2 cascades, 18 application domains, an open‑source ecosystem, and practical resources such as code, a model zoo, and assessment questions.

By Haggai Roitman