arXiv AI
Aug 5
Sphere Retraction Normalizations
arXiv:2608. 02668v1 Announce Type: cross Abstract: Residual connections are the de facto mechanism for training deep neural networks stably.
Inexact calculus of variations on the hyperspherical tangent bundle with connections to the attention mechanism
arXiv:2507. 15431v4 Announce Type: replace Abstract: We offer a theoretical mathematical background through Lagrangian optimization on the unit hyperspherical manifold and its tangential structure.
Related stories
arXiv AI
Aug 12
Riemann GeoResolver: A Non-Euclidean Attention Framework from Euclidean Resolver to Hyperbolic-Spherical Geometry
arXiv:2608. 10416v1 Announce Type: cross Abstract: We present a theoretical foundation for inverse-distance attention, from its Euclidean prototype (Resolver) to its non-Euclidean realization (Riemann GeoResolver).
arXiv Machine Learning
2d ago
Iso-Riemannian Optimization on Learned Data Manifolds
arXiv:2510. 21033v3 Announce Type: replace-cross Abstract: We develop a theory of iso-Riemannian optimization for problems constrained to learned data manifolds, a setting in which classical Riemannian optimization - and Riemannian gradient descent in particular - can be poorly suited.
arXiv Machine Learning
Aug 4
Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design
arXiv:2608. 01283v1 Announce Type: new Abstract: All Transformer-based large language models compute attention via the Euclidean inner product, an architectural choice that Dong et al.
Hugging Face Trending Papers
Jun 11
Different Layers, Different Manifolds: Module-Wise Weight-Space Geometry in Transformer Optimization
Weight-space geometry plays a central role in neural network optimization, yet manifold constraints are often applied uniformly across all weight matrices. In this work, we ask whether different transformer modules prefer different manifold geometries.
arXiv AI
Jun 2
Hyperspherical Variational Autoencoders Using Efficient Spherical Cauchy Distribution
arXiv:2506. 21278v3 Announce Type: replace-cross Abstract: We propose spherical Cauchy (spCauchy) latent variables for variational autoencoders on hyperspherical latent spaces.
arXiv Machine Learning
Jun 9
Barycentric Projections of Optimal Transport Plans on Riemannian Manifolds
arXiv:2606. 07926v1 Announce Type: cross Abstract: Optimal transport couplings are probabilistic objects, while many learning pipelines require deterministic maps.
Hugging Face Trending Papers
Jul 1
Group-Equivariant Poincaré Convolutional Networks
While recent advancements like the Poincaré ResNet have demonstrated the potential of learning visual representations directly in hyperbolic space, their optimisation remains hampered by the computationally intensive nature of Riemannian gradients and the strict boundaries of the manifold. Furthermore, standard hyperbolic networks treat spatial transformations of the same object as distinct hierarchical concepts, leading to redundant parameter usage and vanishing signals.
arXiv AI
Jun 12
Different Layers, Different Manifolds: Module-Wise Weight-Space Geometry in Transformer Optimization
arXiv:2606. 13276v1 Announce Type: cross Abstract: Weight-space geometry plays a central role in neural network optimization, yet manifold constraints are often applied uniformly across all weight matrices.
arXiv Machine Learning
Jul 14
From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers
arXiv:2607. 10677v1 Announce Type: new Abstract: Self-attention is a ubiquitous primitive in modern sequence models, yet its operator-level geometry is only partially understood.
arXiv Machine Learning
Jul 10
Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima
arXiv:2607. 08380v1 Announce Type: new Abstract: An important quantity in the theory of gradient descent (GD) is the \emph{sharpness}, defined as the largest eigenvalue of the objective Hessian.
arXiv AI
Jul 2
Group-Equivariant Poincar\'e Convolutional Networks
arXiv:2607. 00556v1 Announce Type: cross Abstract: While recent advancements like the Poincar\'e ResNet have demonstrated the potential of learning visual representations directly in hyperbolic space, their optimisation remains hampered by the computationally intensive nature of Riemannian gradients and the strict boundaries of the manifold.