arXiv:2607. 08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al.
By Kaihua Ding
arXiv:2610.20417v1 Announce Type: new
Abstract: Audits of large language model (LLM) extraction from full-length documents report severe unreliability, leaving two questions open: what the failures a...
By Jiahong Qian
The study investigates whether a second review by a different large language model (LLM) improves error detection in code, documentation, and analyses. Using 30 artifacts with 150 planted errors and 900 review sessions across three reviewer models, the authors find that a top-tier cross-model reviewer performs similarly to same-model reviews in fresh sessions, identifies partly different errors, and that a combination of a fresh same-model review and a cross-model review detects more errors than two fresh same-model reviews. However, the results do not conclusively separate the effects of model difference from reviewer capability, and a lightweight cross-model reviewer does not outperform same-model review.
whyItMatters":"The findings suggest that while cross-model reviews can enhance error detection, the benefit depends on reviewer skill and the specific review strategy, informing how teams might structure LLM-based code review processes."
By Tae-Eun Song
The paper introduces a portable validity screen for evaluating large language model (LLM) confidence signals, mirroring clinical personality assessment practices. It computes metrics such as the rate of high‑confidence errors (L), low‑confidence correct answers (Fp), and their contrast (RBS), without adding new statistics, to determine whether a signal truly tracks correctness. Applied to 20 frontier models on 452 two‑alternative items and 33 models on 1,500 MMLU items, the screen classifies models as Valid, Indeterminate, or Invalid, revealing that many models fail to meet validity criteria depending on confidence thresholds.
By Jon-Paul Cacioli
AMLworld-Compact introduces an importance‑weighted downsampling protocol for evaluating large language models on AMLworld’s test splits, retaining all illicit transactions while sampling benign ones from supervised‑score strata. The HT‑Coreset method reduces evaluation targets by 271× on HI‑Small and 611× on LI‑Small, providing unbiased confusion‑count estimates with exact recall and ratio estimates for precision and F1. Experiments with boosted‑tree strata and Graph Feature Preprocessor features show lower F1 RMSE compared to uniform sampling, and audit tools reveal that lexical checks do not reliably indicate correctness of model outputs.
By Natapong Nitarach, Phume Ngampornsukswadi, Pittawat Taveekitworachai, Surapon Nonesung, Warit Sirichotedumrong, Kunat Pipatanakul
arXiv:2608. 08029v1 Announce Type: cross Abstract: Khatri et al.
By Alizishaan Khatri, Dun Li Chan