The paper introduces TPR-Attention, an attention mechanism that operates over tensor‑product representations to embed structured inductive bias into deep learning models. Experiments on compositional tasks demonstrate that TPR‑Attention outperforms existing architectural components in achieving combinatorial generalization. The results suggest that incorporating explicit compositional structure into neural attention can improve systematic generalization.
By Melisa Civeleko\u{g}lu, Isabeau Pr\'emont-Schwarz
arXiv:2505.09716v4 Announce Type: replace
Abstract: Out-of-distribution (OOD) generalisation through composition requires a system to discover invariant properties from input-output associations and...
By George Dimitriadis, Spyridon Samothrakis
The paper presents a necessary and sufficient condition for provable compositional generalization in neural networks, identifying two key principles: structural alignment and unambiguous minimized representations. It rigorously proves this condition, verifies it in Lean 4, and demonstrates its applicability in few-shot settings, including the SCAN jump task. The authors also outline an algorithmic approach and illustrate it with a minimal example, all derived purely from mathematical analysis without empirical validation.
By Yuanpeng Li
arXiv:2510.03075v4 Announce Type: replace-cross
Abstract: Compositional generalization, the ability to generate novel combinations of known concepts, is a key ingredient for visual generative models....
By Karim Farid, Rajat Sahay, Yumna Ali Alnaggar, Simon Schrodi, Volker Fischer, Cordelia Schmid, Thomas Brox
The paper investigates how memorizing long‑tail data can aid generalization when combined with a model’s ability to compose features. It demonstrates theoretically that, in a linear setting, memorization plus composition enables correct predictions on rare test examples that involve unseen combinations of long‑tail features. Experiments with neural networks on simple datasets confirm that this effect extends beyond linear models and that a model’s compositional capability depends on its architecture.
By Mo Zhou, Haoyang Ma, Rong Ge
arXiv:2601. 04509v2 Announce Type: replace Abstract: Mixed-integer linear programming (MILP) is a foundational framework for combinatorial optimization across science and engineering, but remains hard to solve at scale due to NP-hardness.
By Peixin Huang, Yaoxin Wu, Yining Ma, Cathy Wu, Wei Zhang, Wen Song
The paper introduces a formal definition of compositional structure based on algorithmic information theory, stating that such structure emerges when combined parts reveal shared patterns not evident when modeled separately. It applies this definition across scales to identify where compositionality arises and then inverts the framework to generate new compositional objects by optimizing over strings and images. The resulting artifacts display clear hierarchical reuse and part‑whole organization, indicating that the definition captures genuine compositionality rather than generic complexity or regularity.
By Eric Elmoznino, Tejas Kasetty, Blaise Ag\"uera y Arcas, Yoshua Bengio, Guillaume Lajoie
arXiv:2505.09716v3 Announce Type: replace-cross
Abstract: Out-of-distribution (OOD) generalisation is considered a hallmark of human and animal intelligence. To achieve OOD through composition, a sys...
By George Dimitriadis, Spyridon Samothrakis
arXiv:2603. 00742v2 Announce Type: replace Abstract: While Adam has long been the ubiquitous default optimizer for deep neural networks, Muon has recently seen rapid adoption due to its superior training speed.
By Sara Dragutinovi\'c, Yedi Zhang, Rajesh Ranganath
The paper presents a unified theoretical framework for analyzing and constructing deep neural networks by explicitly modeling tensor operations, which are often abstracted in existing theories. This framework allows researchers to study how architectural complexity has evolved over the past 40 years and to automatically generate new architectures using novel tensor operations. The authors also release a dataset of over 3,000 higher‑complexity architectures that have not yet been explored, providing a resource for future research.
By Nicholas J. Cooper, Fran\c{c}ois G. Meyer, Michael L. Roberts, Carlos Zapata-Carratal\'a, Lijun Chen, Danna Gurari
arXiv:2607. 23634v1 Announce Type: cross Abstract: Attention enables context modeling via query-key scoring with softmax normalization.
By Rui Wang
The paper introduces two new mathematical task ecosystems, IntWorld and PermWorld, that consist of related integer and permutation tasks. These procedurally generated datasets allow researchers to control the number of tasks and study how task diversity affects neural network training. Initial experiments suggest that greater task variety leads to richer internal representations in models.
By Emily Xia Meng, Xuanyu Yang, Junaid Hasan, Michael Ruofan Zeng, Henry Kvinge