The paper studies matrices built from block‑diagonal factors interleaved with fixed permutations, a structured family useful in deep learning for balancing expressivity and efficiency. By applying Riemannian geometry, the authors determine when this class forms a smooth manifold and develop Riemannian tools for the orthogonal two‑factor case. They propose efficient algorithms that use automatic differentiation, allow parameter sharing, and avoid dense matrix construction, testing them on matrix approximation and fine‑tuning large language models, while also exploring properties of factorizations with more factors.
By Ali Aliev, Maxim Rakhuba
arXiv:2607. 19305v1 Announce Type: cross Abstract: Deep neural networks on manifold-valued representations have attracted growing interest, but many basic components remain tied to specific manifolds, rely on Euclidean approximations, or require costly and numerically fragile geometric operations.
By Chen Ziheng
arXiv:2607. 19305v2 Announce Type: replace-cross Abstract: Deep neural networks on manifold-valued representations have attracted growing interest, but many basic components remain tied to specific manifolds, rely on Euclidean approximations, or require costly and numerically fragile geometric operations.
By Chen Ziheng
arXiv:2606. 12120v1 Announce Type: new Abstract: Low-rank optimal transport (OT) mitigates the quadratic scaling of classical solvers, yet existing approaches rely heavily on first-order mirror-descent updates that require careful hyperparameter tuning and ignore the optimization landscape's curvature.
By Pratik Jawanpuria, Bamdev Mishra
arXiv:2503. 24075v4 Announce Type: replace-cross Abstract: Low-rank optimization problems with sparse simplex constraints involve variables that must satisfy nonnegativity, sparsity, and sum-to-1 conditions, making their optimization particularly challenging due to the interplay between low-rank structures and constraints.
By Flavia Esposito, Andersen Ang
arXiv:2608. 06218v1 Announce Type: cross Abstract: We study Muon, a recently proposed matrix-aware optimization method, in the context of the Stiefel manifold.
By Mikhail Solonko, Molozhavenko Alexander, Maxim Rakhuba
arXiv:2606. 02328v1 Announce Type: new Abstract: We explore Riemannian optimization techniques for rank-factored matrix parameters, targeting contemporary deep learning applications.
By Nicholas Knight
arXiv:2608. 02576v1 Announce Type: new Abstract: We consider optimization problems defined on product spaces of simplices.
By Shashwat Kumar, Arafat Rahman, Anuj Srivastava, P. -A. Absil
arXiv:2607. 08783v1 Announce Type: cross Abstract: Manifold-valued measurements are prevalent in various machine learning tasks.
By Ziheng Chen, Yue Song, Rui Wang, Xiao-Jun Wu, Nicu Sebe
arXiv:2510. 21033v3 Announce Type: replace-cross Abstract: We develop a theory of iso-Riemannian optimization for problems constrained to learned data manifolds, a setting in which classical Riemannian optimization - and Riemannian gradient descent in particular - can be poorly suited.
By Willem Diepeveen, Melanie Weber
arXiv:2509. 07779v2 Announce Type: replace-cross Abstract: We study decentralized online Riemannian optimization over manifolds with possibly positive curvature, going beyond the Hadamard manifold setting.
By Emre Sahinoglu, Shahin Shahrampour
The paper introduces an online framework for functional principal component analysis (FPCA) tailored to multidimensional functional data streams. It models functional principal components with tensor product splines, enforcing smoothness and orthonormality via a penalized approach on a Stiefel manifold. The authors present efficient Riemannian stochastic gradient descent and AdaGrad algorithms, along with a dynamic smoothing parameter tuning strategy based on rolling block validation, and provide asymptotic normality results and confidence intervals for the estimators.
By Muye Nanshan, Nan Zhang, Jiguo Cao