arXiv AI By Siddharth Jain, Venkat N. Vedam

Monitor Accuracy Isn't the Answer: A Decision-Value Framework for Evaluating AI Agent Restart Monitors

Read the original on arXiv AI →

The paper argues that classification accuracy alone is insufficient for evaluating AI agent restart monitors. It introduces a decision‑value framework with a decision‑sufficiency criterion and a finite‑sample decision audit to assess whether restarting an agent is beneficial. Empirical results on SymPy issues show that high accuracy can lead to both gains and losses, and that many traces fail to meet the audit’s certification threshold.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Oct 15

Evidence-First Agentic Verification

arXiv:2610.17724v1 Announce Type: new Abstract: Long-horizon, semi-verifiable agentic tasks often rely on rubric-based evaluation, but the evidence needed to assess these tasks can be spread across t...

By Reza Sanayei, Razvan-Gabriel Dumitru, Sunjie Hou, MohammadHossein Rezaei, Haris Riaz, Robert Vacareanu, Minglai Yang, Ying Liu, Mihai Surdeanu, Daniel Yue Zhang
arXiv Machine Learning
Aug 4

Real-Time Detection and Repair of LLM Agent Failures

arXiv:2608. 02464v1 Announce Type: cross Abstract: LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself.

By Sunny Dubey