arXiv AI
Jun 29
Symmetry-Aware Transformer Training for Automated Planning
arXiv:2508. 07743v2 Announce Type: replace Abstract: While transformers excel in many settings, their application in the field of automated planning is limited.
Toward Manifest Relationality in Transformers via Symmetry Reduction
arXiv:2602. 18948v2 Announce Type: replace Abstract: Transformer models contain substantial internal redundancy arising from coordinate-dependent representations and continuous symmetries, in model space and in head space, respectively.
Related stories
arXiv AI
Jun 17
Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity
arXiv:2606. 17830v1 Announce Type: cross Abstract: Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence.
Hugging Face Trending Papers
Aug 12
Reducing Symmetry Increase in Equivariant Neural Networks
Equivariant Neural Networks (ENNs) have empowered numerous applications in scientific fields. Despite their remarkable capacity for representing geometric structures, ENNs suffer from degraded expressivity when processing symmetric inputs: the output representations are invariant to transformations that extend beyond the input's symmetries.
arXiv AI
Jun 12
Different Layers, Different Manifolds: Module-Wise Weight-Space Geometry in Transformer Optimization
arXiv:2606. 13276v1 Announce Type: cross Abstract: Weight-space geometry plays a central role in neural network optimization, yet manifold constraints are often applied uniformly across all weight matrices.
arXiv Machine Learning
6d ago
Reducing Symmetry Increase in Equivariant Neural Networks
arXiv:2608. 12010v1 Announce Type: new Abstract: Equivariant Neural Networks (ENNs) have empowered numerous applications in scientific fields.
arXiv AI
Jun 4
Platonic Transformers: A Solid Choice For Equivariance
arXiv:2510. 03511v3 Announce Type: replace-cross Abstract: While widespread, Transformers lack inductive biases for geometric symmetries common in science and computer vision.
Hugging Face Trending Papers
Jun 11
Different Layers, Different Manifolds: Module-Wise Weight-Space Geometry in Transformer Optimization
Weight-space geometry plays a central role in neural network optimization, yet manifold constraints are often applied uniformly across all weight matrices. In this work, we ask whether different transformer modules prefer different manifold geometries.
Hugging Face Trending Papers
Jun 24
Communicability-Inspired Positional Encoding (CIPE)
Positional encodings (PEs) are essential for Transformers. Yet designing effective PEs for non-Euclidean graphs remains challenging.
arXiv AI
Jul 29
The Scaling Properties of Implicit Deductive Reasoning in Transformers
arXiv:2605. 04330v2 Announce Type: replace Abstract: We investigate the scaling properties of implicit deductive reasoning over Horn clauses in depth-bounded Transformers.
arXiv AI
Jul 14
Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
arXiv:2607. 11875v1 Announce Type: cross Abstract: We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models.
arXiv Machine Learning
Jun 25
Communicability-Inspired Positional Encoding (CIPE)
arXiv:2606. 25293v1 Announce Type: new Abstract: Positional encodings (PEs) are essential for Transformers.
Hugging Face Trending Papers
Jul 13
Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning.