arXiv AI
Jun 29
Symmetry-Aware Transformer Training for Automated Planning
arXiv:2508. 07743v2 Announce Type: replace Abstract: While transformers excel in many settings, their application in the field of automated planning is limited.
Toward Manifest Relationality in Transformers via Symmetry Reduction
arXiv:2602. 18948v2 Announce Type: replace Abstract: Transformer models contain substantial internal redundancy arising from coordinate-dependent representations and continuous symmetries, in model space and in head space, respectively.
Related stories
arXiv AI
Jun 17
Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity
arXiv:2606. 17830v1 Announce Type: cross Abstract: Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence.
Hugging Face Trending Papers
Aug 12
Reducing Symmetry Increase in Equivariant Neural Networks
Equivariant Neural Networks (ENNs) have empowered numerous applications in scientific fields. Despite their remarkable capacity for representing geometric structures, ENNs suffer from degraded expressivity when processing symmetric inputs: the output representations are invariant to transformations that extend beyond the input's symmetries.
arXiv AI
Jun 12
Different Layers, Different Manifolds: Module-Wise Weight-Space Geometry in Transformer Optimization
arXiv:2606. 13276v1 Announce Type: cross Abstract: Weight-space geometry plays a central role in neural network optimization, yet manifold constraints are often applied uniformly across all weight matrices.
arXiv Machine Learning
Aug 13
Reducing Symmetry Increase in Equivariant Neural Networks
arXiv:2608. 12010v1 Announce Type: new Abstract: Equivariant Neural Networks (ENNs) have empowered numerous applications in scientific fields.
arXiv AI
Jun 4
Platonic Transformers: A Solid Choice For Equivariance
arXiv:2510. 03511v3 Announce Type: replace-cross Abstract: While widespread, Transformers lack inductive biases for geometric symmetries common in science and computer vision.
Hugging Face Trending Papers
Jun 11
Different Layers, Different Manifolds: Module-Wise Weight-Space Geometry in Transformer Optimization
Weight-space geometry plays a central role in neural network optimization, yet manifold constraints are often applied uniformly across all weight matrices. In this work, we ask whether different transformer modules prefer different manifold geometries.
Hugging Face Trending Papers
Jun 24
Communicability-Inspired Positional Encoding (CIPE)
Positional encodings (PEs) are essential for Transformers. Yet designing effective PEs for non-Euclidean graphs remains challenging.
arXiv Machine Learning
Sep 15
Disentangling Representation Evolution in Transformers through Directional Decomposition
arXiv:2609.15975v1 Announce Type: cross Abstract: Transformer representations evolve through learned additive transformations that either preserve their current direction or redirect it. We study thi...
arXiv Machine Learning
2d ago
Parameter symmetries determine representational geometry in overparameterized nonlinear networks
arXiv:2609.39078v1 Announce Type: new Abstract: Representations are routinely used across machine learning, psychology, and neuroscience to draw inferences about the computations of biological and ar...
arXiv AI
Jul 29
The Scaling Properties of Implicit Deductive Reasoning in Transformers
arXiv:2605. 04330v2 Announce Type: replace Abstract: We investigate the scaling properties of implicit deductive reasoning over Horn clauses in depth-bounded Transformers.
arXiv AI
Jul 14
Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
arXiv:2607. 11875v1 Announce Type: cross Abstract: We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models.