The paper presents a unified theoretical framework for analyzing and constructing deep neural networks by explicitly modeling tensor operations, which are often abstracted in existing theories. This framework allows researchers to study how architectural complexity has evolved over the past 40 years and to automatically generate new architectures using novel tensor operations. The authors also release a dataset of over 3,000 higher‑complexity architectures that have not yet been explored, providing a resource for future research.
By Nicholas J. Cooper, Fran\c{c}ois G. Meyer, Michael L. Roberts, Carlos Zapata-Carratal\'a, Lijun Chen, Danna Gurari
The paper discusses tensorizing neural networks by reshaping dense weight matrices into higher-order tensors and approximating them with low-rank tensor network decompositions. This approach offers promising model compression and introduces bond indices that create new latent spaces, potentially enhancing interpretability. Despite encouraging empirical results, tensorized neural networks remain underused, and the authors call for more research to address practical scaling and adoption challenges.
By Safa Hamreras, Sukhbinder Singh, Rom\'an Or\'us
The paper introduces TPR-Attention, an attention mechanism that operates over tensor‑product representations to embed structured inductive bias into deep learning models. Experiments on compositional tasks demonstrate that TPR‑Attention outperforms existing architectural components in achieving combinatorial generalization. The results suggest that incorporating explicit compositional structure into neural attention can improve systematic generalization.
By Melisa Civeleko\u{g}lu, Isabeau Pr\'emont-Schwarz
arXiv:2606. 10913v1 Announce Type: new Abstract: We explore whether intrinsic symmetries of the training data lead to conserved quantities during gradient-flow training of neural networks.
By Jakob Galley, Vahid Shahverdi, Axel Flinth
arXiv:2606. 00130v1 Announce Type: cross Abstract: We study Automatically Differentiable Nonlinear Tensor Networks (ADNTNs), a family of structured weight generators whose compact core tensors are trained end-to-end by reverse-mode automatic differentiation (AD).
By Andrzej Cichocki, Michal Wietczak
The paper introduces Neural Expansion, a geometric framework that explains how deep neural networks generalize by expanding directionally-selective ray coverage in input space. It defines Generalization Intervals (GIs) to measure stable prediction regions around training samples and shows through experiments that training progressively enlarges this coverage, especially along directions with low Jacobian singular values. This expansion accounts for correct test predictions, adversarial robustness, and handling of mislabeled or out-of-distribution data, offering a mechanistic complement to existing global theories.
By Chashi Mahiul Islam, Samuel Jacob Chacko, Mao Nishino, Canlin Zhang, Xiuwen Liu
Systematic generalization remains a significant challenge in deep learning. In particular, combinatorial generalization - generalizing to new configurations of known factors of variation - is effortle...
arXiv:2606. 25975v1 Announce Type: new Abstract: Common first-order optimizers, such as Adam, implicitly treat each parameter block as an unstructured vector, which disregards the multilinear weight structure present in many modern machine learning models.
By Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko, Sergei Kudriashov, Maxim Rakhuba
arXiv:2601.06961v2 Announce Type: replace
Abstract: The success of deep neural networks largely depends on the statistical structure of the training data. While learning dynamics and generalization o...
By Taishi Watanabe, Ryo Karakida, Jun-nosuke Teramae
arXiv:2606. 06772v1 Announce Type: cross Abstract: Understanding the generalization performance of over-parameterized neural networks has become a central topic in deep learning theory.
By Junyu Zhou, Puyu Wang, Yunwen Lei, Marius Kloft, Yiming Ying
arXiv:2607. 01311v1 Announce Type: new Abstract: Deep learning has outgrown any single mathematical explanation.
By Zhilin Zhao
arXiv:2606. 06624v1 Announce Type: new Abstract: In the current era of deep learning and especially generative models, there is significant investment in training very large generative models.
By San Buchanan, Druv Pai, Peng Wang, Yi Ma