← Back to all news
arXiv Machine Learning By Steve Hanneke, Juexiao Wang

The optimal information complexity of VC learning

Read the original on arXiv Machine Learning →

The paper investigates the evaluated Conditional Mutual Information (eCMI) within the Conditional Mutual Information (CMI) framework for learning algorithms. It demonstrates that for VC classes, an optimal PAC guarantee can be achieved by constructing a learning algorithm whose eCMI scales as O(d) in the realizable case, where d is the VC-dimension. The proposed algorithm is a randomized Majority-of-5 base learners that attains an optimal in-expectation generalization guarantee.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 7

Conformal Prediction Sets Quantify Information Gain: A Theoretical Perspective

Conformal prediction sets, which provide finite‑sample coverage guarantees, are examined from an information‑theoretic perspective. The authors introduce a family of generalized information measures that link set size and coverage to entropy, showing that Shannon mutual information can be expressed exactly in terms of these measures. They prove that, in classification tasks, the reduction in set size due to additional information is bounded by calibration‑dependent members of this family and satisfies a data‑processing inequality, and they empirically demonstrate that set‑size reduction and mutual information can rank features differently in greedy selection.

By Kevin Zhang, Stephen Bates
More like this →
arXiv Machine Learning
Oct 16

Does Compression Imply Generalization? A Minimum Description Length Perspective

The paper investigates whether compression explains why over‑parameterized neural networks generalize, using the minimum description length (MDL) principle. It introduces prequential regret, a computable metric from the training trajectory, and shows that in the nearly interpolating regime it upper‑bounds the description length of the trained parameters, leading to a high‑probability generalization bound in the PAC‑MDL setting. The study also connects MDL with the algorithm‑level information bottleneck by bounding encoder‑complexity correction with MDL parameter description length, and demonstrates that prequential regret correlates strongly (Pearson > 0.9) with generalization error on CIFAR‑10 and CIFAR‑100.

By Lincen Yang, Jiayang Shi, Nan Pu, Dani\"el M. Pelt, Matthijs van Leeuwen, Zhong Li, Shujian Yu
More like this →