The paper introduces RADAR, a framework for repairing and auditing confidence scores in large language models (LLMs). It separates confidence repair from deployment decisions and from error auditing, proposing Probe‑Anchored Isotonic Calibration (PAIC) for monotone mapping and a prompt‑clustered kernel test for detecting residual errors. Experiments across checkpoints and benchmarks show that richer predictors can improve confidence estimation, but projecting back to a scalar score can lose gains, and lower prediction risk does not always lead to better review decisions.
By Arijit Dey, Bhartendu Kumar, Ramya Hebbalaguppe, Khuushi Maheshwari
The paper investigates how sequential knowledge editing can degrade a language model’s ability to discern reliable evidence from unreliable evidence without affecting overall accuracy. Using a conservatively tuned LoRA on Qwen2.5‑7B‑Instruct, the authors show that after 1,000 edits the model’s arbitration score for untouched facts drops by 36%, leading to higher error rates on its most confident decisions, while MMLU accuracy remains unchanged. The study also finds that in some model‑method combinations, sequential edits can reduce MMLU to chance levels even though edit success and locality remain perfect.
By Atul Anand
arXiv:2607. 03386v1 Announce Type: new Abstract: Agentic AI systems are increasingly used to edit, refine, and repair decision policies, but evaluating these edits is difficult when per-state expert action labels are unavailable.
By Peiying Zhu, Sidi Chang
arXiv:2610.33262v1 Announce Type: cross
Abstract: We study efficient, adaptive auditing of categorical labels in a fixed dataset, motivated by biodiversity-credit programs in which expert verificatio...
By Fangyuan Lin, Noah Krever, Jakub \v{C}ern\'y, Lily Xu
AMLworld-Compact introduces an importance‑weighted downsampling protocol for evaluating large language models on AMLworld’s test splits, retaining all illicit transactions while sampling benign ones from supervised‑score strata. The HT‑Coreset method reduces evaluation targets by 271× on HI‑Small and 611× on LI‑Small, providing unbiased confusion‑count estimates with exact recall and ratio estimates for precision and F1. Experiments with boosted‑tree strata and Graph Feature Preprocessor features show lower F1 RMSE compared to uniform sampling, and audit tools reveal that lexical checks do not reliably indicate correctness of model outputs.
By Natapong Nitarach, Phume Ngampornsukswadi, Pittawat Taveekitworachai, Surapon Nonesung, Warit Sirichotedumrong, Kunat Pipatanakul
The paper evaluates whether recent LLM-based node classification methods on text‑attributed graphs truly outperform simple baselines. By tuning four baseline models on the same data splits and comparing them to the best reported LLM results, the authors find that while some gains are modest (e.g., Correct & Smooth trailing by 2.3–2.8 percentage points), several reported improvements remain significant. They also highlight evaluation risks such as test‑set hyperparameter tuning, misuse of test‑pool labels, undisclosed baseline reuse, and inconsistencies across papers.
By Michal Podstawski
The paper introduces a method to mitigate reviewer lottery error in scientific peer review by generating checklists of criticisms from historical review data. These checklists are organized into a tree of coarsenings, and the optimal coarsening is chosen to maximize correlated agreement among reviewers. Experiments on NeurIPS 2021 and ICLR 2022 data show that the learned checklists significantly reduce mean squared error in simulated and human review settings.
By Xuanyou Liu, Ziyang Guo, Jason Hartline, Jessica Hullman
arXiv:2608. 12342v1 Announce Type: cross Abstract: Ensuring the accuracy of financial documents is critical for economic analysis, regulatory compliance, and corporate decision-making.
By Ying He, Zhouhong Gu, Zhecheng Hu, Yubo Zhou, Hao Shen, Jiaqing Liang, Zhaoqian Dai, Shuguang Ma, Fei Yu, Yanghua Xiao, Zhixu Li
The paper investigates when it is better to return an existing draft answer or revise it using retrieved evidence in retrieval‑augmented QA systems. By grading both the draft and its candidate revision with the same correctness judge, the authors define a paired effect called recoverability and train policies to predict it before revision. Experiments on 25,870 open‑domain questions show that a recoverability‑based scorer outperforms a draft‑correctness scorer across multiple Llama setups, improving accuracy–revision trade‑offs and closing a significant portion of the oracle gap, though it still applies harmful revisions in a substantial fraction of cases.
By Nicholas Kashani Motlagh, Tim Anderson, Jeremy Gwinnup, Grant Erdmann
The paper investigates how AI systems that perform best‑of‑n search require different validation strategies as the search width changes. It shows that auditing only small search widths leaves a gap in reliability estimates for larger widths, and proposes retaining candidate ranks and truth labels to estimate reliability across all widths up to N. The authors derive theoretical bounds on the minimax mean‑squared error, design procedures that achieve these bounds, and demonstrate that a shared audit can significantly reduce maximum error across many widths in practical CodeRM pools.
By Ricardo Fitas
The paper introduces DISCERN, a two-tier protocol for certifying that updates to production models do not increase risk. It first uses unlabeled data to detect benign updates based on disagreement rates, then selectively labels only disagreements through an anytime-valid confidence sequence. The method achieves finite-sample validity with label-complexity bounds of order ρ²/ε², demonstrating significant label savings and strong empirical performance across 14,000+ audit streams.
By Vishnu Bindu Balachandran
arXiv:2608. 05212v1 Announce Type: new Abstract: Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers.
By Zhixiang Liang, Yifei Liu, Yidan Huang, Haozhe Zhao, Beichen Huang, Jiaqi Wang, Nan Duan, Qiong Cao