arXiv AI
Jun 9
Margin-Adaptive Confidence Ranking for Reliable LLM Judgement
arXiv:2605. 15416v2 Announce Type: replace-cross Abstract: Jung et al.
Localize-Then-Decide Guarantees for LLM Judgments
Related stories
arXiv AI
Aug 19
SCOPE: Selective Conformal Optimized Pairwise LLM Judging
SCOPE is a framework that calibrates an acceptance threshold for large language models used as pairwise judges, ensuring that the error rate among non-abstained judgments does not exceed a user-specified level α. It introduces Bidirectional Preference Entropy (BPE) to provide a bias-neutral uncertainty signal by querying the judge in both response positions and converting the averaged preference probability into an entropy-based score. Across multiple pairwise judging benchmarks, BPE outperforms standard confidence proxies in calibration and discrimination, while SCOPE consistently meets the target risk bound (empirical FDR ≈0.097–0.099 at α=0.10) and retains substantial coverage, accepting up to 2.4× more judgments under the same risk constraint.
arXiv Machine Learning
Jun 15
From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation
arXiv:2606. 13221v2 Announce Type: replace Abstract: Evaluating new large language models typically requires costly human annotation campaigns at scale.
arXiv Machine Learning
Jun 2
How to Correctly Report LLM-as-a-Judge Evaluations
arXiv:2511. 21140v4 Announce Type: replace Abstract: Large language models (LLMs) are widely used as scalable evaluators of model responses in lieu of human annotators.
arXiv AI
Aug 13
Accuracy and Order Sensitivity Diverge Under Label-Free Strategies
arXiv:2608. 11947v1 Announce Type: cross Abstract: Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, which makes them unreliable measures of model knowledge.
arXiv Computation and Language
2d ago
The Score Granularity Gap in Black-Box LLM Classification: A Comparative Study of Confidence Constructions
The paper investigates the "score granularity gap" in black-box large language model (LLM) classifiers, asking how finely a confidence score can be thresholded for deployment. By comparing seven confidence construction methods across 25 model-dataset pairs, the authors find that single-shot verbalized confidence, when properly converted to a probability, ranks well but offers only a few distinct threshold values, limiting operational flexibility. The study also shows that multi-query aggregation can improve weak models but may harm strong ones, and provides concrete guidance for deployment trade-offs.
Hugging Face Trending Papers
Aug 5
Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification
Confidence estimation is essential when LLMs are used for classification, indicating when predictions can be trusted. However, common approaches such as verbalization produce extremely sparse outputs.
arXiv AI
Aug 21
Improved Confidence Estimates for Black-Box Large Language Models
arXiv:2608. 19323v1 Announce Type: cross Abstract: Uncertainty quantification (UQ) is essential for the safe deployment of large language models (LLMs).
arXiv Machine Learning
Jun 5
A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
arXiv:2601. 21817v2 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm.
Hugging Face Trending Papers
Jun 2
SenseJudge: Human-Centric Preference-Driven Judgment Framework
Large Language Models (LLMs) as judges across various scenarios such as assessing model responses is becoming an increasingly accepted paradigm. However, existing judgment approaches often rely on trained judgers using fixed preference data, which tend to overlook diverse user preferences and struggle to adapt to real-world human-AI dialogue scenarios.
arXiv AI
Jul 7
Aligning Language Models with Selective Prediction
arXiv:2607. 03528v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as critical decision-making components in high-stakes real-world AI systems, rendering LLM reliability a foremost practical concern.
arXiv Machine Learning
Aug 4
Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees
arXiv:2608. 02455v1 Announce Type: new Abstract: Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth.