When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals
arXiv:2607. 08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al.
The paper investigates the reliability of ranking tables produced by small-sample evaluations of large language models (LLMs). Using LLM‑inferred prompt structure across eight model variants, the authors find that prompt‑structure recovery is highly unstable, with only the bottom of the ranking consistently reproducible. They demonstrate that standard evaluation practices can misrepresent model performance and propose reporting practices to improve transparency.
arXiv:2607. 08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al.
arXiv:2610.20417v1 Announce Type: new Abstract: Audits of large language model (LLM) extraction from full-length documents report severe unreliability, leaving two questions open: what the failures a...
The study investigates whether a second review by a different large language model (LLM) improves error detection in code, documentation, and analyses. Using 30 artifacts with 150 planted errors and 900 review sessions across three reviewer models, the authors find that a top-tier cross-model reviewer performs similarly to same-model reviews in fresh sessions, identifies partly different errors, and that a combination of a fresh same-model review and a cross-model review detects more errors than two fresh same-model reviews. However, the results do not conclusively separate the effects of model difference from reviewer capability, and a lightweight cross-model reviewer does not outperform same-model review. whyItMatters":"The findings suggest that while cross-model reviews can enhance error detection, the benefit depends on reviewer skill and the specific review strategy, informing how teams might structure LLM-based code review processes."
The paper introduces a portable validity screen for evaluating large language model (LLM) confidence signals, mirroring clinical personality assessment practices. It computes metrics such as the rate of high‑confidence errors (L), low‑confidence correct answers (Fp), and their contrast (RBS), without adding new statistics, to determine whether a signal truly tracks correctness. Applied to 20 frontier models on 452 two‑alternative items and 33 models on 1,500 MMLU items, the screen classifies models as Valid, Indeterminate, or Invalid, revealing that many models fail to meet validity criteria depending on confidence thresholds.
AMLworld-Compact introduces an importance‑weighted downsampling protocol for evaluating large language models on AMLworld’s test splits, retaining all illicit transactions while sampling benign ones from supervised‑score strata. The HT‑Coreset method reduces evaluation targets by 271× on HI‑Small and 611× on LI‑Small, providing unbiased confusion‑count estimates with exact recall and ratio estimates for precision and F1. Experiments with boosted‑tree strata and Graph Feature Preprocessor features show lower F1 RMSE compared to uniform sampling, and audit tools reveal that lexical checks do not reliably indicate correctness of model outputs.
arXiv:2608. 08029v1 Announce Type: cross Abstract: Khatri et al.
arXiv:2609.22478v1 Announce Type: cross Abstract: Behavioural evaluations of hosted language models can vary because the evaluated service, the measurement instrument, or both differ across runs. We...
arXiv:2608. 11423v1 Announce Type: new Abstract: Robust comparisons of federated aggregation methods require joint consideration of predictive performance, threat definitions, metric semantics, and execution provenance.
arXiv:2607. 16122v1 Announce Type: new Abstract: Evaluations should do more than measure a models current performance.
The paper argues that verbalized confidence—once viewed as overconfident and coarse—has become the preferred soft‑scoring method for LLM‑as‑a‑Judge on top‑tier proprietary models released after 2025. Experiments on SummEval, AggreFact, and HelpSteer2 across up to 18 LLMs show that log‑probabilities are no longer the best signal, and that adding an overconfidence advisory and self‑debate further improves calibration and robustness. The authors note that these enhancements incur little accuracy loss on post‑2025 models but do affect pre‑2025 ones, highlighting a compatibility shift in how confidence should be measured.
The paper argues that relying solely on overall accuracy or F1 scores to validate automatic scorers in LLM evaluation can conceal prompt‑specific errors that distort comparative results. By auditing six models on 180 invented‑entity questions across six prompts, the authors show that a scorer’s overall F1 of .916 masks substantial variation—up to 33.3%—in error rates depending on the prompt. In a second audit, differences in measured prompt effects dropped dramatically when multiple raters were used, highlighting the need to validate the exact comparison rather than the scorer alone.
arXiv:2606. 27383v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as research assistants, yet it remains unclear whether they can calibrate research takeaways to the strength and scope of the supporting evidence.