arXiv AI
When Tools Lie: Reliability of Mathematical Agents Under Corrupted Tool Feedback
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
2d ago
VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
The paper introduces VerifierBench, a benchmark comprising 222 tasks from six agent benchmarks and 4,993 hidden test cases designed to evaluate the generation of executable verifiers by large language model agents. It presents Verifier4Verifier (V4V), a six-check audit that screens source verifiers and test suites, and reports that across 19 models, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers more reliably rejecting invalid solutions than accepting valid ones. Case studies highlight that many false rejections stem from verifiers that accept no tested solution, revealing unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions, and underscore the need to test both acceptance of correct solutions and other permissible solutions.
arXiv AI
Aug 5
Designing for Doubt: The Case for Informed Abstention in Autonomous Agents
arXiv:2606. 02965v2 Announce Type: replace Abstract: As large language models gain tool access and are deployed as autonomous agents capable of editing records, executing transactions, and modifying infrastructure, we still evaluate them based on the sole metric of task completion.
arXiv AI
Aug 5
Verified Tool Calls Improve LLM Agent Reliability Under Non-Atomic Failures
arXiv:2608. 02645v1 Announce Type: cross Abstract: Large Language Model (LLM) agents rely on external tools to perform multistage tasks.
arXiv AI
Aug 26
Rebuild Dossier: Mechanically-Enforced Specs for Agentic App Rebuilds, and What Model-Tier Failures Reveal
The paper introduces rebuild‑dossier, an open‑source tool that locks an application’s real interface before code is written and enforces one‑test‑at‑a‑time building through automated checks. In experiments, a compliant agent failed a held‑back test while a rule‑breaking agent passed, showing that a passing test suite can be gamed. The study also demonstrates that the automated check mechanism, rather than interface‑locking alone, is crucial for reliable rebuilds, and that multi‑level verification catches errors that single‑level checks miss.
arXiv AI
Aug 25
Robustness Analysis of Agentic AI to Inconsistent and Incomplete Tool Responses
arXiv:2608.22676v1 Announce Type: new Abstract: Robustness to a bad tool return means answering it in the way that return calls for, which depends on how the tool went wrong. A tool that has failed a...
arXiv AI
Sep 10
Agents Trust Tools Too Much: Measuring Reliance on Unreliable Tools
arXiv:2609.05587v1 Announce Type: new Abstract: Existing evaluations of tool-using agents primarily measure whether an agent can successfully complete diverse tasks with tools. These evaluations gene...