arXiv AI By Yukun Zhang, Kemu Xu, Yishen Chen

How Do Agent Harnesses Create Value? Planning Information and Release Control in Stateful LLM Agents

Read the original on arXiv AI →

The paper investigates how agent harnesses—specifically planning guidance, execution organization, and completion verification—affect performance in retail and airline pilot tasks. By comparing fixed, task‑specific plans to shuffled policy text of equal length, the study finds that fixed plans improve success rates by about 7 percentage points, especially on complex tasks. A read‑only verifier rejects a majority of invalid episodes while incurring minimal cost, and its impact varies with the penalty for erroneous acceptance, often matching the full planning‑plus‑verification benefit at a lower cost.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
6d ago

From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings

The paper introduces Ask2Decide, a benchmark of 43 planning tasks designed to test whether large language model (LLM) agents can identify and request missing decision‑relevant information from humans. It collects 79 interaction sessions from 19 participants, showing that human replies often omit critical facts, and uses these data to calibrate a simulated operator. Evaluation of four state‑of‑the‑art LLMs reveals that their initial plans are infeasible on 58‑98% of tasks when information is missing, and that agents only slightly outperform random guessing in distinguishing useful from irrelevant facts.

By Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui, Ao Qu, Ou Zheng, Jiangbo Yu, Haris N. Koutsopoulos, Jinhua Zhao, Yuan Zhou
arXiv Machine Learning
Oct 19

Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers

The paper introduces SPLIT, a method that optimally allocates test‑time compute among sampling, model selection, and verification for language models and agents. Using an exact probability formula that accounts for task difficulty, shared blind spots, and verifier correlations, SPLIT is fitted via empirical Bayes on a small labeled pilot and then used to plan budgets for unseen tasks. Across seven diverse benchmarks, 26 models, and six verifier types, SPLIT achieves near‑optimal performance, outperforming rule‑based baselines by up to 2.5 percentage points and providing insights into when additional LLM judges reduce precision.

By Srinivasan Iyengar
arXiv AI
6d ago

Veritas: A Verifier Generation Harness for Agent Tasks

Veritas is a harness that automatically generates verifiers for agent tasks, adding an evidence header, a library of domain rules, and authoring principles refined through a self‑improvement loop. It was evaluated on six benchmarks, outperforming seven other harnesses on all four execution‑scored tasks and matching or exceeding performance on pairwise comparison tasks. The system also improves agreement with official verdicts across multiple model families and harness‑benchmark pairs.

By Daoan Zhang, Qinbo Bai, Dehai Min, Xingyu Mu, Ziyi Chen, Jianing Yin