The paper presents a unified theoretical framework for analyzing and constructing deep neural networks by explicitly modeling tensor operations, which are often abstracted in existing theories. This framework allows researchers to study how architectural complexity has evolved over the past 40 years and to automatically generate new architectures using novel tensor operations. The authors also release a dataset of over 3,000 higher‑complexity architectures that have not yet been explored, providing a resource for future research.
By Nicholas J. Cooper, Fran\c{c}ois G. Meyer, Michael L. Roberts, Carlos Zapata-Carratal\'a, Lijun Chen, Danna Gurari
The paper discusses tensorizing neural networks by reshaping dense weight matrices into higher-order tensors and approximating them with low-rank tensor network decompositions. This approach offers promising model compression and introduces bond indices that create new latent spaces, potentially enhancing interpretability. Despite encouraging empirical results, tensorized neural networks remain underused, and the authors call for more research to address practical scaling and adoption challenges.
By Safa Hamreras, Sukhbinder Singh, Rom\'an Or\'us
The paper introduces TPR-Attention, an attention mechanism that operates over tensor‑product representations to embed structured inductive bias into deep learning models. Experiments on compositional tasks demonstrate that TPR‑Attention outperforms existing architectural components in achieving combinatorial generalization. The results suggest that incorporating explicit compositional structure into neural attention can improve systematic generalization.
By Melisa Civeleko\u{g}lu, Isabeau Pr\'emont-Schwarz
arXiv:2606. 10913v1 Announce Type: new Abstract: We explore whether intrinsic symmetries of the training data lead to conserved quantities during gradient-flow training of neural networks.
By Jakob Galley, Vahid Shahverdi, Axel Flinth
arXiv:2606. 00130v1 Announce Type: cross Abstract: We study Automatically Differentiable Nonlinear Tensor Networks (ADNTNs), a family of structured weight generators whose compact core tensors are trained end-to-end by reverse-mode automatic differentiation (AD).
By Andrzej Cichocki, Michal Wietczak
The paper introduces Neural Expansion, a geometric framework that explains how deep neural networks generalize by expanding directionally-selective ray coverage in input space. It defines Generalization Intervals (GIs) to measure stable prediction regions around training samples and shows through experiments that training progressively enlarges this coverage, especially along directions with low Jacobian singular values. This expansion accounts for correct test predictions, adversarial robustness, and handling of mislabeled or out-of-distribution data, offering a mechanistic complement to existing global theories.
By Chashi Mahiul Islam, Samuel Jacob Chacko, Mao Nishino, Canlin Zhang, Xiuwen Liu