The paper compares a non‑generative verifier (Jev) with a frontier LLM (Astra) on 495 human‑labelled claims from LLM‑AggreFact. Both models achieve statistically indistinguishable accuracy, yet Jev is 187‑fold cheaper and admits more unsupported claims. A confidence‑gated cascade that escalates uncertain claims improves cost‑efficiency, though the benefit comes mainly from the escalation share rather than the probability estimates.
By Alden Do Rosario, Kirollos Atef
The paper investigates how persistent memory in AI agents can lead to over‑trust in stale facts, creating a "Memory Trust Gap" that worsens as model capability increases. Using a benchmark with Benefit and Safety suites across Qwen3 models of varying sizes, the authors show that larger models are more prone to harmful over‑trust, especially when metadata is absent or misleading. They also demonstrate that mitigation strategies such as exposing metadata or pre‑resolving conflicts improve accuracy, but the effectiveness depends on model size and dataset.
By Jundong Hu, Shekar Ramachandran
arXiv:2606.07834v2 Announce Type: replace-cross
Abstract: Evidence-grounded fact verification increasingly uses LLM judges to turn evidence into terminal verdicts. Many task contracts deliberately in...
By Haoran Xu
The paper investigates how agents that inherit consolidated memory can mistakenly rely on constraints that were once true but have since been superseded by newer records. By modeling supersession explicitly and limiting verification to two records, the study shows that native allocation often leads to stale-consistent decisions, while reallocating one verification slot to the critical provenance path markedly improves consistency. The results suggest that memory systems may need separate freshness or supersession signals beyond relevance to avoid such errors.
By Kazuki Nakayashiki
arXiv:2609.25054v1 Announce Type: new
Abstract: For a long-horizon LLM agent, the memory question is not what was once recorded but what \emph{currently holds}. Most designs answer it only indirectly...
By Bowen Qin, Yao Lu
A language model's memory can be worse than having no memory at all. Give a model a memory that kept a wrong conclusion but dropped the work behind it, and it emits that stale value as a confident answer; give the same model an empty memory and it abstains.
arXiv:2606. 25449v1 Announce Type: cross Abstract: A language model's memory can be worse than having no memory at all.
By Alex Kwon
The paper introduces a framework for evaluating how large language model agents revise their success criteria after failures, defining five non‑compensatory conditions that must be met for a criterion revision to be considered valid. Using the CMB‑0.1 protocol, the authors test twelve cross‑domain scenarios across four system configurations, finding that no model trial satisfies all five conditions and highlighting specific failure modes such as zero‑state reconstruction and inadequate intervention sensitivity. They propose a more stringent trace‑anchored CMB‑0.4 protocol to better isolate and measure criterion revision in future studies.
By Guodong Xu
The paper reports on a series of experiments examining how different forms of directives—such as record pointers, criteria, or combinations—affect an agent’s choice of archived source records when it inherits six one-line memories. Across twelve registered studies involving 14,760 attempts on a single instrument lineage, the authors measured the impact of various directive formats on six direct-provider models, nine OpenRouter-served models, and several Claude and Opus 5 models, noting differences in performance metrics and replication outcomes. The results are purely descriptive, detailing the effects of exact edits on fixed panels with registered intervals and no claim of underlying mechanisms.
By Kazuki Nakayashiki
The paper investigates how machine‑learning models can determine whether a claim (a test assertion) remains valid after a code change. It compares two questioning strategies: asking whether a diff preserves behavior versus asking whether a specific claim still holds. The authors find that the latter approach yields far higher precision (up to 0.974) across models of varying cost, while the former performs poorly (precision 0.291–0.329). They also benchmark against a regression‑test selector, showing that even near‑complete knowledge of a change’s reach does not reliably identify falsified claims. The study is grounded in 10,369 mined claims with 184 execution‑verified flips from 23 Python libraries.
By Atul Anand
arXiv:2606. 15474v1 Announce Type: new Abstract: Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down.
By Yitao Li
arXiv:2608.31016v1 Announce Type: cross
Abstract: Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission: information the encounter established that the n...
By Sebastian Fox, Luke Markham, Ryan Lail, Michael Karotsieris