The study examines whether accuracy gaps between source and target tasks can certify the failure of scalar recalibration maps for large language model judges. Across thirteen judges, two generators, eight domains, and 1,176 transfers, the accuracy gap only provides a weak lower bound on target calibration error and can predict opposite outcomes. Even with a finite‑sample lower certificate, the method shows low power (0.13) and does not reliably indicate when recalibration will fail.
arXiv:2606. 15474v1 Announce Type: new Abstract: Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down.
ProxyGuard is a new method for assessing the reliability of randomized data release mechanisms that use shared target sets. It offers two modes: named-release mode, which corrects for multiplicity and certifies specific releases, and direct shared-target mode, which evaluates independent mechanism draws on a common target, providing finite-sample reliability guarantees without needing independent target batches. In a registered study, direct mode increased power from 5.6% to 64.2% at a 0.95 reliability level, while named mode performed better under high-signal evidence.
arXiv:2606. 29654v1 Announce Type: new Abstract: Multi-agent deliberation among LLMs can improve reasoning, but deployment requires deciding when the current answer is reliable enough to act on and when it should be escalated to human review.
arXiv:2608. 14617v1 Announce Type: cross Abstract: A recurring proposal in legal AI is to improve case-outcome prediction by fusing uncertainty tools (evidence graphs with belief propagation, sequential Bayesian odds updating, Dempster-Shafer combination, and conformal prediction) into one pipeline.
arXiv:2604. 18245v3 Announce Type: replace Abstract: Large language models operate in protocols containing multiple calls, yet added calls are usually evaluated only by their net effect.
The paper reports a preregistered audit of language‑model judges used as measurement instruments, revealing that the assumption that a model’s responses remain stable over time is invalid. Across nearly 53,000 audited requests, repeat rankings and byte‑identical replays fell far below required reliability thresholds, with three identified mechanisms—label‑to‑meaning bias, candidate gaps below the noise floor, and input permutation noise—explaining the discrepancy. The study proposes a three‑level snapshot‑identity framework, eight design rules, and a reporting checklist to prevent such reliability failures in future evaluations.
arXiv:2608. 07914v1 Announce Type: new Abstract: Behavioral contamination detectors can return "no evidence" either because a benchmark is clean or because the audit has little power.
The paper investigates the reliability of language‑model judges used as measurement instruments on shared endpoints. Through two preregistered audits of 52,988 requests, the authors found that repeat rankings and byte‑identical replays fell far short of required thresholds, revealing significant instability. They identify three mechanisms—label‑to‑meaning bias, candidate gaps below the noise floor, and input permutation noise—that explain the gap, and propose a snapshot‑identity ladder, design rules, and a reporting checklist to mitigate such failures.
arXiv:2607. 08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al.
arXiv:2606. 29484v1 Announce Type: cross Abstract: Modern deepfake detectors are rarely consumed as bare classifiers.
arXiv:2608. 00915v1 Announce Type: new Abstract: Uplift modeling (conditional-average-treatment-effect estimation) drives personalized targeting, yet published uplift benchmarks frequently disagree on which estimator performs best; we show the disagreement is substantially about metrics, not models.
arXiv:2607. 08535v1 Announce Type: cross Abstract: An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed.