The paper introduces Product Interactions, an algebraic framework that unifies modern neural architectures by treating data as algebraic elements and multiplication as a core operation. It shows how convolutional, equivariant, state‑space, and attention models arise from different orders of product interactions and symmetry constraints, and demonstrates that a simplified algebraic dynamical layer can outperform Mamba on CIFAR‑10. The framework positions architecture design as a systematic algebraic construction problem, enabling both analysis of existing models and generation of new ones.
arXiv:2606. 19538v1 Announce Type: new Abstract: Convolutional networks, recurrent networks, and transformers each encode different inductive biases -- locality, sequential memory, and content-dependent pairwise interaction -- and have remained mathematically distinct since their inception.
arXiv:2610.17424v1 Announce Type: cross Abstract: Respecting a-priori known symmetries of underlying data is a key principle in designing efficient neural architectures. Convolutional neural networks...
CrossGMN introduces a graph metanetwork that processes a trained source network and an initialized target network simultaneously, enabling equivariant cross‑architecture weight‑space transformations. By preserving symmetry through cross‑network message passing, CrossGMN can refine target network initializations while remaining invariant to source permutations and equivariant to target permutations. Experiments demonstrate that CrossGMN accelerates knowledge distillation, transfers across datasets without retraining, and unifies compression from diverse source architectures into a common target architecture.
The paper introduces the Feature Interaction Modeling Operator (FM-Operator), a point‑wise query neural operator that explicitly designs feature construction and multiplicative interactions between sensor observations and query coordinates. By reinterpreting the DeepONet aggregation as a diagonally constrained multiplicative interaction, FM-Operator restructures both feature construction and interaction to enable richer information exchange while maintaining point‑wise evaluation. Experiments on several PDE benchmarks show that FM-Operator consistently outperforms vanilla DeepONet and improves upon the Shift‑DeepONet baseline, indicating that tailored representation construction and interaction can enhance query‑based neural operator performance.
The paper introduces the Universal Weight Encoder (UWE), a scalable representation for weight-space learning that uses a typed graph of interfaces and modules to respect parameter symmetries across diverse neural network architectures. UWE combines local equivariant updates with global attention over invariant interface summaries, enabling a single encoder to process multiple supported architectures efficiently. Experiments show UWE matches or surpasses architecture‑specific state‑of‑the‑art methods on standard benchmarks and excels on a large zoo of 1,200 transformers, ranking unseen architectures accurately while using significantly less memory.
arXiv:2505.09716v4 Announce Type: replace Abstract: Out-of-distribution (OOD) generalisation through composition requires a system to discover invariant properties from input-output associations and...
arXiv:2608. 06839v1 Announce Type: new Abstract: Artificial Neural networks (ANNs) are often treated as black-box models, making explainability a central challenge in deep learning.
arXiv:2607. 06634v1 Announce Type: new Abstract: Compact networks built from Clifford algebra Cl(3,0) primitives are exactly SO(3)-equivariant and learn synthetic 3D vector laws from few samples.
arXiv:2604. 07242v3 Announce Type: replace Abstract: Despite deep learning models running well-defined mathematical functions, we lack a formal mathematical framework for describing model architectures.
arXiv:2602. 01083v2 Announce Type: replace Abstract: Weight-space learning studies neural architectures that operate directly on the parameters of other neural networks.
arXiv:2605. 18106v3 Announce Type: replace-cross Abstract: A striking geometric disparity has long persisted in the practice of deep learning.
Additive models buy interpretability by forbidding feature interactions, a constraint that neural instantiations enforce architecturally. We introduce the quadrilateral loss, a differentiable penalty that treats additivity as a measurable behavior instead: a second-order mixed difference on pairs of training points swapping one coordinate, which vanishes if and only if the coordinate carries no interaction, remains informative for piecewise-linear networks, and equals in expectation the per-coordinate interaction mass of the interventional Shapley-GAM.