arXiv:2607. 02104v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise.
arXiv:2608. 15046v1 Announce Type: new Abstract: A fraction of a point of benchmark accuracy is the usual evidence that a compressed model is equivalent to its original.
The paper presents a framework for deriving sharp, label‑free calibration certificates for query‑conditioned probabilistic predictors. It establishes minimax lower bounds on calibration residuals under various betting restrictions, shows that these bounds can be discontinuous and that Lipschitz stakes yield stable, sharp bounds. The authors also demonstrate practical algorithms—projected online gradient ascent and anytime‑valid betting—for generating fixed‑horizon and anytime‑valid certificates, and validate the approach on a preregistered study of four predictors across two datasets.
arXiv:2608. 07913v1 Announce Type: cross Abstract: Selective-risk certificates promise that accepted outputs meet a declared error target.
arXiv:2607. 11920v1 Announce Type: cross Abstract: Evaluating decisions made under uncertainty is hard when labeled outcomes are scarce, costly, or confounded with luck.
The study analyzes 373,019 judgments from LLM‑scored benchmarks, decomposing variance into system, item, judge, and interaction components via generalizability theory. It finds that with a single judge, generalizability converges to a ceiling determined by the system‑by‑judge variance, which is substantially lower in pairwise preference settings, allowing one judge to suffice. The research also reveals significant biases in presentation order and highlights that many published win‑rate claims fall below the measured floor of the benchmarks.
arXiv:2610.09541v1 Announce Type: cross Abstract: Signals that predict whether a chain-of-thought (CoT) trace is correct are compared by AUC, but deploying one requires a threshold with a guarantee....
arXiv:2608.21496v1 Announce Type: cross Abstract: AI systems increasingly generate alternatives, inspect evidence, and deploy a selected output. Validation is therefore target-relative: evidence cert...
arXiv:2606. 08517v1 Announce Type: new Abstract: Selective predictors answer on confident inputs and abstain elsewhere; deploying one safely needs a single finite-sample certificate that simultaneously upper-bounds the selected risk, lower-bounds the acceptance probability $\pacc$ above a floor $\pmin$, and lower-bounds the deployment utility.
arXiv:2608. 06762v1 Announce Type: new Abstract: Bisimulation metrics quantify behavioral similarity in Markov decision processes, but their Wasserstein fixed-point operator updates every state pair and incurs quadratic pairwise work.
The paper investigates how many human annotators are equivalent to a panel of 32 large‑language‑model (LLM) judges. By comparing the panel’s label distributions to empirical human labels on three ChaosNLI tasks, the authors find two distinct effective panel sizes: distribution‑error matching yields effective sizes of 2.304, 3.750, and 3.445, while spectral matching gives 4.242, 6.459, and 6.499, indicating a 1.72–1.89× gap. The study also explores how spectral diversity, participation ratio, and panel composition affect effective size, and demonstrates that carefully chosen panels can outperform baseline accuracy while improving effective size.
The paper introduces a method for certifying the quality of solutions produced by heuristic solvers for general‑sum extensive‑form games. By relating the Nash gap of a weighted average strategy to external regret and a signed correction term, the authors derive an online certificate that can be computed in linear time per checkpoint. Experiments on the biased Leduc family show that selecting the iterate with the smallest certificate from a portfolio of training‑free regret‑matching solvers achieves high success rates across multiple test sets.