arXiv AI
Jun 9
Margin-Adaptive Confidence Ranking for Reliable LLM Judgement
arXiv:2605. 15416v2 Announce Type: replace-cross Abstract: Jung et al.
Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification
Confidence estimation is essential when LLMs are used for classification, indicating when predictions can be trusted. However, common approaches such as verbalization produce extremely sparse outputs.
Related stories
arXiv AI
Jul 7
Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language
arXiv:2607. 03882v1 Announce Type: cross Abstract: LLMs are increasingly deployed as post-hoc explainers of AI-generated outputs, yet it remains unclear whether they can reliably communicate probabilistic information in natural language.
arXiv Machine Learning
Jun 5
Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference
arXiv:2606. 05308v1 Announce Type: new Abstract: With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set.
arXiv Machine Learning
Jun 2
How to Correctly Report LLM-as-a-Judge Evaluations
arXiv:2511. 21140v4 Announce Type: replace Abstract: Large language models (LLMs) are widely used as scalable evaluators of model responses in lieu of human annotators.
arXiv AI
Jun 2
Gradients with Respect to Semantics Preserving Embeddings Tell the Uncertainty of Large Language Models
arXiv:2605. 04638v2 Announce Type: replace-cross Abstract: Uncertainty quantification (UQ) is an important technique for ensuring the trustworthiness of LLMs, given their tendency to hallucinate.
arXiv AI
Jul 24
ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
arXiv:2607. 20526v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in settings where fluent but incorrect answers can be costly.
arXiv Machine Learning
Jun 29
CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes
arXiv:2606. 20820v2 Announce Type: replace Abstract: Can we trust evaluation scores to capture an LLM's true real-world performance?
arXiv Machine Learning
Jun 29
Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings
arXiv:2606. 27997v1 Announce Type: new Abstract: Benchmarks of machine learning models often include many datasets, making evaluation expensive.
arXiv Machine Learning
Jun 15
From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation
arXiv:2606. 13221v2 Announce Type: replace Abstract: Evaluating new large language models typically requires costly human annotation campaigns at scale.
arXiv AI
Aug 11
Embedding Trust: Semantic Isotropy Predicts Nonfactuality in Long-Form Text Generation
arXiv:2510. 21891v2 Announce Type: replace-cross Abstract: To deploy large language models (LLMs) in high-stakes application domains that require substantively accurate responses to open-ended prompts, we need reliable, computationally inexpensive methods that assess the trustworthiness of long-form responses generated by LLMs.
arXiv AI
Jul 7
Evaluating LLM Uncertainty in Long-Form Generation Using Deterministic Ground Truth
arXiv:2607. 03870v1 Announce Type: new Abstract: As LLMs generate increasingly long outputs, effective uncertainty estimation must identify errors at fine-grained levels rather than discard entire responses.
Hugging Face Trending Papers
Jul 7
SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generation
Uncertainty estimation is essential not only for the trustworthy deployment of large language models (LLMs) but also as a foundation for self-refinement in LLM generation. However, existing approaches operate at suboptimal granularities: token-level scores lack semantic coherence, while sequence-level scores fail to localize errors.