Conformal prediction sets, which provide finite‑sample coverage guarantees, are examined from an information‑theoretic perspective. The authors introduce a family of generalized information measures that link set size and coverage to entropy, showing that Shannon mutual information can be expressed exactly in terms of these measures. They prove that, in classification tasks, the reduction in set size due to additional information is bounded by calibration‑dependent members of this family and satisfies a data‑processing inequality, and they empirically demonstrate that set‑size reduction and mutual information can rank features differently in greedy selection.
Conformal prediction is a popular tool for uncertainty quantification that outputs prediction sets with finite-sample coverage guarantees. While prediction set size is commonly used as a heuristic mea...
The paper investigates whether compression explains why over‑parameterized neural networks generalize, using the minimum description length (MDL) principle. It introduces prequential regret, a computable metric from the training trajectory, and shows that in the nearly interpolating regime it upper‑bounds the description length of the trained parameters, leading to a high‑probability generalization bound in the PAC‑MDL setting. The study also connects MDL with the algorithm‑level information bottleneck by bounding encoder‑complexity correction with MDL parameter description length, and demonstrates that prequential regret correlates strongly (Pearson > 0.9) with generalization error on CIFAR‑10 and CIFAR‑100.
arXiv:2505. 19033v2 Announce Type: replace-cross Abstract: Conformal prediction (CP) is a widely used frequentist framework to quantify uncertainty by constructing prediction sets with user-specified marginal coverage guarantees.
A central question in modern learning theory is what notion of compression, if any, explains why over-parameterized neural networks generalize. Existing answers measure complexity at different levels,...
The paper investigates the evaluated Conditional Mutual Information (eCMI) within the Conditional Mutual Information (CMI) framework for learning algorithms. It demonstrates that for VC classes, an optimal PAC guarantee can be achieved by constructing a learning algorithm whose eCMI scales as O(d) in the realizable case, where d is the VC-dimension. The proposed algorithm is a randomized Majority-of-5 base learners that attains an optimal in-expectation generalization guarantee.
The paper extends conformal prediction by introducing a geometric view of uncertainty through credal sets, which are families of probability distributions compatible with CP regions. It provides analytical characterisations and efficient algorithms for common credal uncertainty measures, reducing complexity from exponential to linear after sorting p-values. The authors also unify these measures with classical conformal summaries using Tsallis entropy penalties, prove an impossibility theorem for split-conformal regression, and demonstrate through experiments that the new summaries recover ranking information lost by prediction-set size, enabling finer-grained uncertainty assessment.
arXiv:2610.27402v1 Announce Type: cross Abstract: Motivated by the notion of features from machine learning, we study the complexity of representing data that allows for evaluation with a given famil...
arXiv:2407. 12288v5 Announce Type: replace-cross Abstract: The progress of machine learning over the past decade is undeniable.
arXiv:2604. 23716v3 Announce Type: replace Abstract: Information-theoretic (IT) measures are ubiquitous in artificial intelligence: entropy drives decision-tree splits and uncertainty quantification, cross-entropy is the default classification loss, mutual information underpins representation learning and feature selection, and transfer entropy reveals directed influence in dynamical systems.
arXiv:2607. 06570v1 Announce Type: cross Abstract: Value-of-information (VOI) analysis is usually conducted under a single probability measure.
arXiv:2606. 28598v1 Announce Type: cross Abstract: Prediction sets should have high coverage to be useful, but some coverage notions are more practically relevant than others.
arXiv:2509. 04631v2 Announce Type: replace Abstract: Transductive conformal prediction addresses the simultaneous prediction for multiple data points.