arXiv AI
Aug 7
An Optimal Agnostic PAC Algorithm
arXiv:2608. 06363v1 Announce Type: cross Abstract: Let $H\subseteq\{-1,+1\}^X$ be a class of finite VC dimension $d\ge1$.
The optimal information complexity of VC learning
The paper investigates the evaluated Conditional Mutual Information (eCMI) within the Conditional Mutual Information (CMI) framework for learning algorithms. It demonstrates that for VC classes, an optimal PAC guarantee can be achieved by constructing a learning algorithm whose eCMI scales as O(d) in the realizable case, where d is the VC-dimension. The proposed algorithm is a randomized Majority-of-5 base learners that attains an optimal in-expectation generalization guarantee.
Related stories
arXiv Machine Learning
Oct 7
Conformal Prediction Sets Quantify Information Gain: A Theoretical Perspective
Conformal prediction sets, which provide finite‑sample coverage guarantees, are examined from an information‑theoretic perspective. The authors introduce a family of generalized information measures that link set size and coverage to entropy, showing that Shannon mutual information can be expressed exactly in terms of these measures. They prove that, in classification tasks, the reduction in set size due to additional information is bounded by calibration‑dependent members of this family and satisfies a data‑processing inequality, and they empirically demonstrate that set‑size reduction and mutual information can rank features differently in greedy selection.
arXiv Machine Learning
Oct 16
Does Compression Imply Generalization? A Minimum Description Length Perspective
The paper investigates whether compression explains why over‑parameterized neural networks generalize, using the minimum description length (MDL) principle. It introduces prequential regret, a computable metric from the training trajectory, and shows that in the nearly interpolating regime it upper‑bounds the description length of the trained parameters, leading to a high‑probability generalization bound in the PAC‑MDL setting. The study also connects MDL with the algorithm‑level information bottleneck by bounding encoder‑complexity correction with MDL parameter description length, and demonstrates that prequential regret correlates strongly (Pearson > 0.9) with generalization error on CIFAR‑10 and CIFAR‑100.
arXiv Machine Learning
Jun 25
Bellman-sufficient Information Complexity
arXiv:2606. 11171v4 Announce Type: replace Abstract: We develop Bellman-sufficient information complexity, a formal representation-level framework for sequential decision making.
Hugging Face Trending Papers
Oct 15
Does Compression Imply Generalization? A Minimum Description Length Perspective
A central question in modern learning theory is what notion of compression, if any, explains why over-parameterized neural networks generalize. Existing answers measure complexity at different levels,...
arXiv Machine Learning
Sep 30
Preferent Compression Bounds Are Tight
arXiv:2609.36030v1 Announce Type: new Abstract: The lack of rigorous safety and performance certificates remains a key bottleneck to the deployment of modern learning-based methods. Sample compressio...
arXiv Machine Learning
Aug 14
The Optimal Sample Complexity of Multiclass and List Learning
arXiv:2604. 24749v2 Announce Type: replace Abstract: While the optimal sample complexity of binary classification in terms of the VC dimension is well-established, determining the optimal sample complexity of multiclass classification has remained open.
arXiv Statistics ML
Oct 26
Combinatorial Characterizations of the Learnability of Conditional Distributions
arXiv:2610. 31890v1 Announce Type: new Abstract: We study the PAC-learnability of function classes that map inputs to probability distributions over a finite space.
arXiv Machine Learning
Aug 4
Optimal Unambiguous DNFs and Alon-Saks-Seymour
arXiv:2608. 02533v1 Announce Type: cross Abstract: We construct unambiguous DNFs having width $O(n)$ but $0$-certificate complexity $\Omega(n^2)$.
arXiv Machine Learning
Oct 8
An Accuracy--Information Tradeoff for Loss-Difference Conditional Mutual Information
arXiv:2610. 09206v1 Announce Type: new Abstract: Loss-difference conditional mutual information (ld-CMI) uses the smallest of the standard observations in the supersample hierarchy of generalization bounds: it measures what a learner's loss differences reveal about which candidate of each pair it was trained on.
Hugging Face Trending Papers
Oct 6
Conformal Prediction Sets Quantify Information Gain: A Theoretical Perspective
Conformal prediction is a popular tool for uncertainty quantification that outputs prediction sets with finite-sample coverage guarantees. While prediction set size is commonly used as a heuristic mea...
Hugging Face Trending Papers
Aug 3
Optimal Unambiguous DNFs and Alon-Saks-Seymour
We construct unambiguous DNFs having width $O(n)$ but $0$-certificate complexity $Ω(n^2)$. By utilizing the special structure of these DNFs, we prove a lifting theorem with a constant-sized gadget that lifts the DNF to a communication problem, while losslessly translating the separation in certificate complexity to a separation in communication complexity.