The paper investigates whether compression explains why over‑parameterized neural networks generalize, using the minimum description length (MDL) principle. It introduces prequential regret, a computable metric from the training trajectory, and shows that in the nearly interpolating regime it upper‑bounds the description length of the trained parameters, leading to a high‑probability generalization bound in the PAC‑MDL setting. The study also connects MDL with the algorithm‑level information bottleneck by bounding encoder‑complexity correction with MDL parameter description length, and demonstrates that prequential regret correlates strongly (Pearson > 0.9) with generalization error on CIFAR‑10 and CIFAR‑100.
By Lincen Yang, Jiayang Shi, Nan Pu, Dani\"el M. Pelt, Matthijs van Leeuwen, Zhong Li, Shujian Yu
Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization.
arXiv:2610. 39027v1 Announce Type: cross Abstract: What computational role does overparameterisation play during learning?
By Ryan Singh, Miguel De Llanza, Christopher L. Buckley
arXiv:2607. 11883v1 Announce Type: new Abstract: Compression is fundamental to intelligence.
By Shikai Qiu, Marc Finzi, Yujia Zheng, Kun Zhang, Andrew Gordon Wilson
arXiv:2606. 04834v1 Announce Type: new Abstract: Minimum Description Length (MDL) formalizes the principle of Occam's razor by optimizing the total description length: $L(\mathrm{model})+L(\mathrm{data} \ | \ \mathrm{model})$.
By Qian Li, Xinyu Mao, Shang-Hua Teng, Guangxu Yang
arXiv:2606. 11171v2 Announce Type: replace Abstract: We develop indexed Bellman information complexity, a representation-level theory of interactive decision making centered on information indices and reference histories.
By Yunbei Xu
arXiv:2505. 23869v4 Announce Type: replace-cross Abstract: A proposition that connects randomness and compression is put forward via Gibbs entropy over set of measurement vectors associated with a compression process.
By M. S\"uzen
arXiv:2602. 01903v2 Announce Type: replace Abstract: This work studies online episodic tabular Markov decision processes (MDPs) with known transitions and develops best-of-both-worlds algorithms that achieve refined data-dependent regret bounds in the adversarial regime and variance-dependent regret bounds in the stochastic regime.
By Mingyi Li, Taira Tsuchiya, Kenji Yamanishi
The paper presents quantitative generalization bounds for fixed‑precision transformer models, addressing both in‑distribution statistical generalization and length generalization. It derives guarantees based on model parameters, quantization space, and alphabet size, and shows how training length and sample size trade off, with longer training being exponentially more efficient. Experiments confirm the theoretical predictions.
By Marius Henry, Michael Rizvi-Martel, Michael Hahn
arXiv:2407. 12288v5 Announce Type: replace-cross Abstract: The progress of machine learning over the past decade is undeniable.
By Hong Jun Jeon, Benjamin Van Roy
The paper argues that Large Language Models (LLMs) do not function as Solomonoff induction estimators because their training objectives—cross‑entropy, negative log‑likelihood, and next‑token prediction—optimize fit to a supplied conditional distribution rather than a program‑weighted universal mixture. It further contends that additional computation alone does not transform these models into optimal predictors without external hyper‑parameter or architectural changes. The authors suggest that neurosymbolic machine learning, exemplified by models such as Fable and Astra, represents a shift toward symbolic model synthesis, moving beyond purely statistical LLMs.
By Hector Zenil, Abicumaran Uthamacumaran, Luan Ozelim
Conformal prediction sets, which provide finite‑sample coverage guarantees, are examined from an information‑theoretic perspective. The authors introduce a family of generalized information measures that link set size and coverage to entropy, showing that Shannon mutual information can be expressed exactly in terms of these measures. They prove that, in classification tasks, the reduction in set size due to additional information is bounded by calibration‑dependent members of this family and satisfies a data‑processing inequality, and they empirically demonstrate that set‑size reduction and mutual information can rank features differently in greedy selection.
By Kevin Zhang, Stephen Bates