arXiv AI
Clean Scores, Buried Evidence, and Confident Wrong: A Receipt-Based Audit of Frontier Agentic QA
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
3d ago
Beyond fabrication: Misattributed numbers in papers written by AI research agents
The paper examines how AI‑written research papers often contain misattributed numerical claims. By re‑analyzing 540 judgments from a benchmark of nine AI agents on fifteen research problems, the authors manually labeled 256 claims and found that most failures stem from misalignment between the paper and the underlying work rather than outright cheating. They propose a deterministic claim‑to‑evidence checker that achieves over 90% precision and, when combined with an LLM verifier, offers a more reliable audit of AI‑generated research.
arXiv AI
Oct 7
A Trust Layer for Agent Evaluation
The paper introduces a Trust Layer for Agent Evaluation, a post‑hoc framework that appends a belief flag to each benchmark score. It checks four properties: alignment with the benchmark’s grading logic, traceable computation of a passing answer, consistency between the agent’s claim and actual behavior, and stability across repeated runs. Applied to five agent configurations on 108 tasks, the study finds that only 22.6% of recorded passes satisfy all checks, revealing widespread issues such as untraceable computation, false completion claims, and unstable results.
arXiv AI
Oct 16
ClaimScope: Label-Blind Witness Verification for Mathematical Claim Auditing
ClaimScope is a 72‑task benchmark that audits mathematical claims by requiring a verdict, scope, counterexample, and repair, covering areas such as optimizer‑state transport and exact $2 imes2$ linear algebra. The study evaluates Qwen3‑8B using a witness checker that only examines the model’s submitted state, showing that verifier feedback improves counterexample acceptance and safe completions compared to generic feedback. The results demonstrate that separating claim classification from executable evidence can yield gains in witness‑structure correction without necessarily improving broader reasoning.
arXiv AI
Oct 19
PRIME: Towards Verifiable Evidence-Grounded Reasoning in LLM Agents
PRIME (Precommitted Rules for Inference from Minimal Evidence) is a framework designed to improve the reliability of large language model agents when verifying records against organizational requirements. It predefines evidence routes for each possible conclusion, records check results in a ledger, and refuses any conclusion that is not fully justified by the recorded evidence. In experiments on 480 public‑record tasks, PRIME eliminated unjustified conclusions entirely and increased the average score for justified conclusions from 30.1 to 59.2 out of 100.
arXiv AI
Aug 25
AUDITA: certified auditing and causal attribution of adverse outcomes in autonomous multi-agent systems
arXiv:2608.22160v1 Announce Type: new Abstract: Physical automation is scaling toward fleets of embodied machines commanded by an AI brain. Early deployments already run factories and warehouses at p...
arXiv AI
Sep 30
Agentic Commerce Bench: Measuring Fraud Detection for Agents That Spend Money
The paper introduces the Agentic Commerce Bench (ACB), a benchmark for measuring fraud in AI agents that autonomously spend money. It presents a taxonomy of agentic commerce fraud, a dataset of twenty fraud classes derived from real production data, and an open‑source detector stack called gordonguard for auditing and replaying hostile counterparties. The study shows that current reasoning layers and security scanners perform poorly on many classes, highlighting the need for better detection mechanisms.