The paper introduces a penalty‑aware evaluation framework for Retrieval‑Augmented Generation (RAG) systems that uses asymmetric scoring, knowledge‑gap canaries, and a failure‑attribution pipeline. Applying this framework to three commercial RAG products and a baseline on SimpleQA‑Verified, the authors find that while overall accuracy is similar across systems, canary violation rates vary dramatically, showing that systems differ more in when they answer than in what they answer. The study demonstrates that penalty‑aware scoring can reorder system rankings and is robust across different penalty settings.
By Alden Do Rosario, Hussein Younes, Felipe Pires
The paper argues that evaluating continual knowledge‑updating methods solely at a final checkpoint and a single adapter rank can be misleading. By fixing a periodic hierarchy and comparing it to cumulative replay on a 24‑month Wikidata stream, the authors show that the apparent best method changes depending on the evaluation month, replay LoRA rank, and query formulation. They recommend reporting performance trajectories and capacity sweeps, and only declaring a robust winner when the ranking remains stable across the evaluation region.
By Heejin Choi
The paper investigates intrinsic self‑correction, where a language model revises its own answer without new evidence. Across 29 open‑weight LLMs on BoolQ, GSM8K, and Corr2Cause, the study tracks how revisions change correctness, revealing that while some models improve significantly, others lose a notable fraction of correct answers. The authors compare three runtime strategies—keeping the initial answer, always accepting the revision, and selectively gating revisions—and find that the best approach depends on the model and task, suggesting that self‑correction should be treated as a revision policy rather than a uniformly beneficial second pass.
By Tianzhu Zhang
The paper investigates how sequential knowledge editing can degrade a language model’s ability to discern reliable evidence from unreliable evidence without affecting overall accuracy. Using a conservatively tuned LoRA on Qwen2.5‑7B‑Instruct, the authors show that after 1,000 edits the model’s arbitration score for untouched facts drops by 36%, leading to higher error rates on its most confident decisions, while MMLU accuracy remains unchanged. The study also finds that in some model‑method combinations, sequential edits can reduce MMLU to chance levels even though edit success and locality remain perfect.
By Atul Anand
DynaKRAG is a unified framework that learns a state‑conditioned policy to control evidence acquisition in multi‑hop retrieval‑augmented generation. It uses a deterministic validity layer to build an action set, a learned continuation gate to decide between generating an answer or gathering more evidence, and an advantage scorer to rank evidence operations by predicted gain. Across HotpotQA, 2Wiki, and MuSiQue with various backbone models, DynaKRAG achieves top EM and F1 scores, improves token and retrieval efficiency, and enables terminal evidence compression that reduces context size while boosting answer quality.
By Chenyu Zhou, Yaqi Wu, Xiaolei Guo, Jiaqi Huang, Xianfa Zhang, Junxu Zhang, Zhuo Yu, Zhubo Shi, Jianghao Lin, Dongdong Ge
ReDraft is a reference‑driven revision method for continual post‑training of large multimodal language models. It uses the model’s own incorrect outputs as references, revises them, verifies the revisions, and fine‑tunes on the accepted ones, thereby combining explicit supervision with policy proximity. On tasks such as Counting, Clock Reading, and Jigsaw, ReDraft outperforms standard supervised fine‑tuning and on‑policy methods, achieving higher target‑task gains while dramatically reducing forgetting.
By Zhihao Zhang, Mingqi Wu, Qiaole Dong, Enyu Zhou, Shuo Li, Boyang Liu, Jiazheng Zhang, Honglin Guo, Xin Guo, Shaofan Liu, Junzhe Wang, Dingwei Zhu, Zhiheng Xi, Minlong Peng, Yuan Hua, Qi Zhang, Tao Gui, Xuanjing Huang
arXiv:2609.36587v1 Announce Type: new
Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a prominent approach for improving language-model performance on reasoning tasks using...
By Yupeng Chang, Wenxuan Zhang, Yuan Wu
The paper introduces "stale‑document poisoning," a temporal alignment failure where outdated retrieval evidence causes retrieval‑augmented generation models to produce incorrect answers even when the model could answer correctly without retrieval. A benchmark of 317 verified knowledge reversals across medicine, law, software, and platform policy shows that outdated evidence flips 30–37% of Llama and Qwen answers, rising to 66–75% when models are explicitly instructed to trust the document. The study demonstrates that providing explicit validity information and a recency‑aware re‑ranker can substantially reduce poisoning, highlighting the need for models to assess whether retrieved evidence remains applicable.
By Md Shamim Ahmed, Lukas Galke Poech, Richard R\"{o}ttger
The paper investigates how memory systems can answer a current query correctly yet fail to retain distinctions needed for later updates. Using a paired‑history audit, the authors evaluate 24 history pairs across six synthetic mechanisms and two model backends, achieving perfect reveal accuracy on DeepSeek and high accuracy on GLM. Record‑level audits reveal specific failures in structured reveal memories and frontier late‑reference adequacy, and the authors test a label‑equivariant repair that only partially restores correctness.
By Guangzhe Zhang
arXiv:2609.08279v1 Announce Type: cross
Abstract: Agent memory systems must discard stored information when their history exceeds a fixed token budget. Existing budget-accuracy frontiers quantify the...
By Chen Shen
The paper introduces a joint fact‑verification score that evaluates both answers and the evidence submitted with them. On the FEVEROUS dataset, replacing the DCUF evidence with UnifEE evidence improves the strict score by about 9.6 percentage points, while answer accuracy rises only 1.96 points. The study also shows that increasing context length for large language models yields modest evidence‑gain improvements, and that detailed answer‑evidence analyses uncover patterns missed by aggregate metrics.
By Han Chen, Yingrui Li
arXiv:2608. 08944v1 Announce Type: cross Abstract: A failed retrieval-augmented generation (RAG) answer can be consistent with several unseen responses to evidence repair.
By Wenzhang Du