arXiv AI By Daoan Zhang, Qinbo Bai, Dehai Min, Xingyu Mu, Ziyi Chen, Jianing Yin

Veritas: A Verifier Generation Harness for Agent Tasks

Read the original on arXiv AI →

Veritas is a harness that automatically generates verifiers for agent tasks, adding an evidence header, a library of domain rules, and authoring principles refined through a self‑improvement loop. It was evaluated on six benchmarks, outperforming seven other harnesses on all four execution‑scored tasks and matching or exceeding performance on pairwise comparison tasks. The system also improves agreement with official verdicts across multiple model families and harness‑benchmark pairs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Oct 5

VERSE: Verified Self-Evolving Optimizer for Agent Harnesses

The paper introduces VERSE, a Verified Self‑Evolving optimizer that enhances LLM agent harnesses by allowing the optimizer to test edits, replay failures, and perturb steps while tracking fixes and regressions. VERSE builds its own tools for failure analysis, verification, training audits, and workflow control, and uses this feedback to revise the harness’s prompts, skills, tools, hooks, and notes without changing model weights. In experiments across five executors and multiple languages, VERSE improves all evaluated harness optimizers, achieving higher accuracy on held‑out and out‑of‑distribution tasks compared to the strongest baselines.

By Zekai Wang, Yingqiang Ge, Zekun Wang, Hai Wang, Yuhui Xu, Joshua Frandsen, Shancong Fu, Ashia C. Wilson, Chandan K. Reddy
arXiv AI
Oct 9

Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents

The paper introduces a method for co-evolving inspectable verifiers alongside self-improving agents. By synthesizing verifiers from clustered failures and selecting them based on agreement with a reference set, the authors demonstrate improved held‑out agreement on MBPP+ and outperform a bare LLM judge. They also show that removing anchor guards collapses the verifier into a vacuous grader, yet the collapsed verifier still trains skills effectively, indicating that downstream task scores cannot certify a self‑evolved verifier.

By Xing Zhang, Guanghui Wang, Yanwei Cui, Ziyuan Li, Wei Qiu, Bing Zhu, Peiyang He
arXiv AI
Oct 19

EvalResearchBench: Can AI Agents Build Their Own Feedback for Recursive Self-Improvement?

EvalResearchBench (ERB) is a benchmark that tests whether AI agents can autonomously design evaluation tasks for recursive self‑improvement. In ERB, a researcher agent selects or synthesizes tasks, builds graders, and refines them through pilot tests before producing a final evaluator that runs within fixed time and API budgets. Experiments with nine researchers and thirteen candidate models show that the best automated evaluators achieve about 75% concordance with target benchmarks, below the 91% ceiling set by target disagreements, and that human‑designed task samples still outperform many automated approaches.

By Yaolun Zhang, Tianyi Xu, Yujie Zhao, Jishen Zhao, Qingyun Wu, Huazheng Wang