The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
The paper investigates whether compression explains why over‑parameterized neural networks generalize, using the minimum description length (MDL) principle. It introduces prequential regret, a computable metric from the training trajectory, and shows that in the nearly interpolating regime it upper‑bounds the description length of the trained parameters, leading to a high‑probability generalization bound in the PAC‑MDL setting. The study also connects MDL with the algorithm‑level information bottleneck by bounding encoder‑complexity correction with MDL parameter description length, and demonstrates that prequential regret correlates strongly (Pearson > 0.9) with generalization error on CIFAR‑10 and CIFAR‑100.
Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization.
arXiv:2610. 39027v1 Announce Type: cross Abstract: What computational role does overparameterisation play during learning?
arXiv:2607. 11883v1 Announce Type: new Abstract: Compression is fundamental to intelligence.
arXiv:2606. 04834v1 Announce Type: new Abstract: Minimum Description Length (MDL) formalizes the principle of Occam's razor by optimizing the total description length: $L(\mathrm{model})+L(\mathrm{data} \ | \ \mathrm{model})$.
arXiv:2606. 11171v2 Announce Type: replace Abstract: We develop indexed Bellman information complexity, a representation-level theory of interactive decision making centered on information indices and reference histories.