arXiv AI
Aug 5
Sphere Retraction Normalizations
arXiv:2608. 02668v1 Announce Type: cross Abstract: Residual connections are the de facto mechanism for training deep neural networks stably.
Inexact calculus of variations on the hyperspherical tangent bundle with connections to the attention mechanism
arXiv:2507. 15431v4 Announce Type: replace Abstract: We offer a theoretical mathematical background through Lagrangian optimization on the unit hyperspherical manifold and its tangential structure.
Related stories
arXiv Machine Learning
Aug 20
The Diffusion-Attention Connection
arXiv:2604. 09560v2 Announce Type: replace Abstract: Softmax attention is the row-normalized operator of a diffusion map: both normalize a learned score into a Markov operator, and differ only in what the score is allowed to contain.
arXiv AI
Aug 12
Riemann GeoResolver: A Non-Euclidean Attention Framework from Euclidean Resolver to Hyperbolic-Spherical Geometry
arXiv:2608. 10416v1 Announce Type: cross Abstract: We present a theoretical foundation for inverse-distance attention, from its Euclidean prototype (Resolver) to its non-Euclidean realization (Riemann GeoResolver).
arXiv Machine Learning
Sep 17
Generalizing Adam to Manifolds for Efficiently Training Transformers
The paper introduces a novel generalization of the Adam optimizer to manifold settings, specifically targeting homogeneous spaces such as the Stiefel, symplectic Stiefel, and Grassmann manifolds. By exploiting a global tangent space representation (the Lie subspace), the authors eliminate the need for projection steps and enable all Adam operations to be performed directly on these manifolds. The new optimizer is applied to train transformers and a symplectic autoencoder, achieving orthogonality constraints to machine precision and outperforming existing methods.
arXiv Machine Learning
Aug 18
Iso-Riemannian Optimization on Learned Data Manifolds
arXiv:2510. 21033v3 Announce Type: replace-cross Abstract: We develop a theory of iso-Riemannian optimization for problems constrained to learned data manifolds, a setting in which classical Riemannian optimization - and Riemannian gradient descent in particular - can be poorly suited.
arXiv Machine Learning
Sep 21
Beyond Gaussian Worlds: Latent Geometry Matters for JEPAs
The paper extends the analysis of Joint-Embedding Predictive Architectures (JEPAs) beyond Euclidean latent spaces to Riemannian manifolds. It shows that when latent variables lie on a sphere and the target distribution matches this spherical geometry, every optimal representation recovers the latent state up to an orthogonal transformation, demonstrating that Gaussian uniqueness is not universal. Experiments confirm that geometrically compatible targets improve linear recovery, especially in high-dimensional toroidal settings.
arXiv Machine Learning
Aug 4
Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design
arXiv:2608. 01283v1 Announce Type: new Abstract: All Transformer-based large language models compute attention via the Euclidean inner product, an architectural choice that Dong et al.
Hugging Face Trending Papers
Jun 11
Different Layers, Different Manifolds: Module-Wise Weight-Space Geometry in Transformer Optimization
Weight-space geometry plays a central role in neural network optimization, yet manifold constraints are often applied uniformly across all weight matrices. In this work, we ask whether different transformer modules prefer different manifold geometries.
arXiv AI
Jun 2
Hyperspherical Variational Autoencoders Using Efficient Spherical Cauchy Distribution
arXiv:2506. 21278v3 Announce Type: replace-cross Abstract: We propose spherical Cauchy (spCauchy) latent variables for variational autoencoders on hyperspherical latent spaces.
arXiv Machine Learning
Sep 23
When Riemann flows with Wasserstein: Generative Modeling of Probability Distributions on Manifolds
arXiv:2609.25659v1 Announce Type: new Abstract: Many scientific datasets, such as molecular conformational ensembles or single-cell tissue measurements, are naturally modeled as meta-distributions: d...
arXiv Machine Learning
Jun 9
Barycentric Projections of Optimal Transport Plans on Riemannian Manifolds
arXiv:2606. 07926v1 Announce Type: cross Abstract: Optimal transport couplings are probabilistic objects, while many learning pipelines require deterministic maps.
arXiv Machine Learning
Sep 24
hyperbolix: Hyperbolic Deep Learning in JAX
hyperbolix is an open‑source library for hyperbolic deep learning in JAX, built on Flax NNX. It provides six manifolds—including Euclidean, Poincaré ball, hyperboloid, κ‑stereographic, mixed‑curvature product, and proper velocity space—through a common interface, and implements a wide range of layer families (linear, convolution, attention, normalization, positional encoding, regression, vector quantization). The library also supplies Riemannian optimizers, wrapped distributions, dimensionality‑reduction techniques, and precision‑tested operations that replace numerically unstable formulas on the hyperboloid, ensuring accurate float32 computations at large distances.