arXiv:2610. 39027v1 Announce Type: cross Abstract: What computational role does overparameterisation play during learning?
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
The paper investigates whether compression explains why over‑parameterized neural networks generalize, using the minimum description length (MDL) principle. It introduces prequential regret, a computable metric from the training trajectory, and shows that in the nearly interpolating regime it upper‑bounds the description length of the trained parameters, leading to a high‑probability generalization bound in the PAC‑MDL setting. The study also connects MDL with the algorithm‑level information bottleneck by bounding encoder‑complexity correction with MDL parameter description length, and demonstrates that prequential regret correlates strongly (Pearson > 0.9) with generalization error on CIFAR‑10 and CIFAR‑100.
A central question in modern learning theory is what notion of compression, if any, explains why over-parameterized neural networks generalize. Existing answers measure complexity at different levels,...
arXiv:2607. 20652v1 Announce Type: cross Abstract: Language models are thought to exhibit the phenomenon of superposition, representing many more features than dimensions in their residual streams.
The paper presents a tight generalization bound for deterministic neural networks that relies on the quality of their last hidden layer representations rather than traditional parameter-space metrics. By using a parameter‑free k‑nearest‑neighbor classifier to assess representation quality, the bound decomposes into three measurable components—local class geometry, decoder‑network agreement, and training stability—providing both a non‑vacuous certificate and a diagnostic tool for understanding generalization. Experiments show the bound achieves an 8.0‑percentage‑point gap to test error on CIFAR‑10 with ResNet‑18, yields the first non‑vacuous certificates on CIFAR‑100, and certifies 1.4 % on CIFAR‑10 and 23.1 % on 1,000‑class ImageNet using frozen DINOv2 representations.
arXiv:2610.27402v1 Announce Type: cross Abstract: Motivated by the notion of features from machine learning, we study the complexity of representing data that allows for evaluation with a given famil...
arXiv:2606. 15219v1 Announce Type: new Abstract: In this work, we tackle the following question: Can neural networks trained with gradient-based methods achieve the optimal computational-statistical tradeoff in learning Gaussian single-index models?