arXiv AI
Output Discipline Sets the Floor: A Falsification-Guided, Model-Agnostic Agent Protocol for Auditable Hypothesis-Driven Decision Making
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 28
Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation
arXiv:2607. 23386v1 Announce Type: new Abstract: We document a failure class in frontier large language models -- exception chain collapse -- observed in eligibility evaluation under nested conditional rules of the form "A is required UNLESS B applies, UNLESS C overrides B".
arXiv AI
Aug 24
Calibrating Criterion Revision in LLM Agents: Failure Modes and a Trace-Anchored Protocol
The paper introduces a framework for evaluating how large language model agents revise their success criteria after failures, defining five non‑compensatory conditions that must be met for a criterion revision to be considered valid. Using the CMB‑0.1 protocol, the authors test twelve cross‑domain scenarios across four system configurations, finding that no model trial satisfies all five conditions and highlighting specific failure modes such as zero‑state reconstruction and inadequate intervention sensitivity. They propose a more stringent trace‑anchored CMB‑0.4 protocol to better isolate and measure criterion revision in future studies.
arXiv AI
Oct 16
Auditing Escalation Reliability in Healthcare LLM Agents: A Perturbation-Controlled Benchmark of Workflow-State Failures
The paper introduces a benchmark of 400 synthetic healthcare workflow episodes to evaluate large language model (LLM) agents’ reliability in escalating care when evidence is missing or conflicting. It finds that while clean decision accuracy is high (0.97), models miss a large proportion of necessary escalations (32–97%) under uncertainty, and even the largest 9.7B model’s performance drops to 0.78 in conflict scenarios. A workflow-aware controller improves utility and reduces missed escalations, but its effectiveness is limited by lexical generalization failures.
arXiv AI
Aug 7
Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalzied Financial Agents
arXiv:2608. 06108v1 Announce Type: new Abstract: Investment competence is inherently personalized: the same market evidence can justify different actions for investors with different goals, horizons, portfolios, and risk boundaries.
arXiv AI
Oct 2
From Proposal to Verified Effect: Praxa, an Evidence-Bound Harness for Governed AI Agent Execution
Praxa is an evidence‑bound harness for governed AI agent execution that explicitly represents states such as proposal, authority, dispatch, verified external effect, and promotion through deterministic admission, brokered execution, external read‑back, reconciliation, and reviewed promotion. The authors report four evidence lanes: a repository‑local audit passing all unit and Workerd tests; a pilot on 12 curated tasks where both baseline and reliability‑layer arms passed 17 of 36 trials; a coordination‑proxy comparison where both baseline and a source‑authored candidate completed all 180 trials with equal accuracy but the candidate used fewer tokens and steps; and deployed source/configuration evidence showing bounded reflection, recall accounting, memory compilation, and tool‑health paths. None of the evidence demonstrates superiority in security, safety, or user benefit. whyItMatters:"Praxa provides a testable architecture that makes authority‑to‑effect transitions explicit, offering a framework for verifying AI agent behavior, though current evidence does not prove improved security or performance."
arXiv AI
Aug 12
Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents
arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.