arXiv Machine Learning
Jul 15
Efficient Sequential Calibration with $O(T^{2/3-\epsilon})$ Error Bound
arXiv:2607. 12928v1 Announce Type: new Abstract: We study the online binary sequential calibration problem.
Truthful Calibration Measures for Sequential Prediction
The paper investigates the feasibility of exact truthfulness in calibration measures for sequential binary prediction. It proves that exact truthfulness cannot coexist with completeness and soundness, even when outcomes are independent. The authors then provide two reductions that transform any base calibration measure into additively or multiplicatively approximately truthful ones, achieving a multiplicative truthfulness guarantee that improves upon previous results.
Related stories
arXiv Machine Learning
Jun 9
Proper Calibeating
arXiv:2605. 26703v2 Announce Type: replace-cross Abstract: The classic concept of "calibrated forecasts" and its more recent refinement, "calibeating," are defined with respect to the standard quadratic scoring rule.
arXiv AI
Aug 12
How to Verify Consistency of Probabilistic Claims
arXiv:2608. 11181v1 Announce Type: cross Abstract: When a probabilistic predictor answers many conditional-probability queries, are its answers self-consistent, and can this be verified in polynomial time?
arXiv AI
Sep 10
How to Verify Probabilistic Consistency of Predictive Models
The paper presents an interactive probabilistically checkable proof (PCP) protocol that allows a polynomial‑time verifier to check the approximate consistency of a probabilistic predictor defined by two circuits, P and Q. By evaluating these circuits at a few points and querying a proof oracle that encodes a witnessing probability distribution, the verifier can confirm that the predictor’s many conditional‑probability claims are self‑consistent. The authors also establish that the problem of verifying l₂‑approximate consistency for explicit probabilistic claims lies in NP, with certificates of size O(mn + log B), and show how to eliminate dependence on the input bit‑precision B through a small additive gap.
arXiv Machine Learning
Jun 10
Can we trust our models? Epistemic calibration in second-order classification
arXiv:2606. 10777v1 Announce Type: new Abstract: Uncertainty estimation is critical for deploying machine learning models in high-stakes settings.
arXiv Machine Learning
4d ago
A Ranking Approach for Measuring Calibration
The paper introduces rankECE, a new metric for assessing calibration error in predictive models. Unlike the widely used Expected Calibration Error (ECE), rankECE compares predictions with neighboring probability values, offering theoretical guarantees and empirical evidence that it better approximates ECE than traditional binned methods.
arXiv Machine Learning
Jun 3
Hierarchies of Calibration: Classification meets Regression
arXiv:2606. 03245v1 Announce Type: cross Abstract: Concepts of calibration formalize the compatibility between probabilistic predictions and the respective outcomes.
arXiv Machine Learning
1d ago
Breaking the $T^{2/3}$ Barrier for Sequential Calibration
arXiv:2406.13668v4 Announce Type: replace Abstract: A set of probabilistic forecasts is calibrated if each prediction of the forecaster closely approximates the empirical distribution of outcomes on...
arXiv Computation and Language
Aug 25
Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models
arXiv:2608.22483v1 Announce Type: new Abstract: Large Language Models (LLMs) increasingly support decision-making in high-stakes domains, but they often hallucinate and express confidence that is mis...
arXiv Machine Learning
Aug 17
Sequence prediction under a lying oracle
arXiv:2608. 14102v1 Announce Type: new Abstract: We consider the problem of sequential prediction of an $m$-ary sequence, where at each epoch, (i) the environment selects an outcome from an $m$-ary alphabet, (ii) the learner selects a probability distribution over the same alphabet (unaware of the outcome generated by the environment), and finally, (iii) the learner incurs a cost that depends on the probability assigned to the outcome.
arXiv Statistics ML
Sep 10
A Unifying Perspective on Probabilities as Model Predictions
arXiv:2609.09855v1 Announce Type: new Abstract: Although probabilistic statements are ubiquitous, foundational disagreements persist about their understanding, as exemplified by debates between Bayes...
arXiv Machine Learning
Jul 9
Optimal Conformal Prediction under Epistemic Uncertainty
arXiv:2505. 19033v2 Announce Type: replace-cross Abstract: Conformal prediction (CP) is a widely used frequentist framework to quantify uncertainty by constructing prediction sets with user-specified marginal coverage guarantees.