The paper introduces TPR-Attention, an attention mechanism that operates over tensor‑product representations to embed structured inductive bias into deep learning models. Experiments on compositional tasks demonstrate that TPR‑Attention outperforms existing architectural components in achieving combinatorial generalization. The results suggest that incorporating explicit compositional structure into neural attention can improve systematic generalization.
Systematic generalization remains a significant challenge in deep learning. In particular, combinatorial generalization - generalizing to new configurations of known factors of variation - is effortle...
arXiv:2505.09716v4 Announce Type: replace Abstract: Out-of-distribution (OOD) generalisation through composition requires a system to discover invariant properties from input-output associations and...
The paper presents a necessary and sufficient condition for provable compositional generalization in neural networks, identifying two key principles: structural alignment and unambiguous minimized representations. It rigorously proves this condition, verifies it in Lean 4, and demonstrates its applicability in few-shot settings, including the SCAN jump task. The authors also outline an algorithmic approach and illustrate it with a minimal example, all derived purely from mathematical analysis without empirical validation.
The paper investigates how memorizing long‑tail data can aid generalization when combined with a model’s ability to compose features. It demonstrates theoretically that, in a linear setting, memorization plus composition enables correct predictions on rare test examples that involve unseen combinations of long‑tail features. Experiments with neural networks on simple datasets confirm that this effect extends beyond linear models and that a model’s compositional capability depends on its architecture.
arXiv:2601. 04509v2 Announce Type: replace Abstract: Mixed-integer linear programming (MILP) is a foundational framework for combinatorial optimization across science and engineering, but remains hard to solve at scale due to NP-hardness.
arXiv:2510.03075v4 Announce Type: replace-cross Abstract: Compositional generalization, the ability to generate novel combinations of known concepts, is a key ingredient for visual generative models....
The paper introduces Higher-Order Modular Attention (HOMA), a new attention mechanism that combines standard pairwise self‑attention with an explicit triadic attention pathway. HOMA uses overlapping blocks, local windows, and a low‑rank projection to make triadic interactions tractable. Experiments on controlled PARITY and MATCH3 tasks, as well as TAPE benchmarks, show that HOMA matches or outperforms matched pairwise and purely triadic baselines, especially when dependencies extend beyond triadic order, and it often converges faster and uses parameters more efficiently.
The paper introduces two new mathematical task ecosystems, IntWorld and PermWorld, that consist of related integer and permutation tasks. These procedurally generated datasets allow researchers to control the number of tasks and study how task diversity affects neural network training. Initial experiments suggest that greater task variety leads to richer internal representations in models.
The paper presents a unified theoretical framework for analyzing and constructing deep neural networks by explicitly modeling tensor operations, which are often abstracted in existing theories. This framework allows researchers to study how architectural complexity has evolved over the past 40 years and to automatically generate new architectures using novel tensor operations. The authors also release a dataset of over 3,000 higher‑complexity architectures that have not yet been explored, providing a resource for future research.
arXiv:2602. 24264v2 Announce Type: replace-cross Abstract: Compositional generalization, the ability to recognize familiar parts in novel contexts, is a defining property of intelligent systems.
arXiv:2109. 13445v3 Announce Type: replace-cross Abstract: The capability of Deep Neural Networks (DNNs) to recognize objects in orientations outside the distribution of the training data is not well understood.
arXiv:2505.09716v3 Announce Type: replace-cross Abstract: Out-of-distribution (OOD) generalisation is considered a hallmark of human and animal intelligence. To achieve OOD through composition, a sys...