arXiv AI
Jul 29
Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning
arXiv:2607. 25299v1 Announce Type: cross Abstract: Optimization over the Stiefel manifold plays a significant role in various machine learning tasks.
Geometry-Aware Bayesian Parameter-Efficient Fine-Tuning on the Stiefel Manifold via Stein Variational Gradient Descent
The paper introduces a geometry-aware Bayesian fine‑tuning method that uses Stein variational gradient descent on the Stiefel manifold. By transporting low‑rank adapter matrices along this manifold, the approach preserves orthogonality constraints and yields multiple inference solutions, enabling uncertainty quantification. Experiments demonstrate improved model calibration and higher prediction accuracy compared to Euclidean‑space SVGD and related methods.
Related stories
arXiv Machine Learning
Jun 30
BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning
arXiv:2606. 29184v1 Announce Type: new Abstract: While Low-rank adaptation (LoRA) enables highly efficient fine-tuning by constraining task-specific updates to fixed low-rank subspaces, this rigid design limits representational flexibility and often results in overconfident predictions and miscalibrated uncertainty, especially in low-data regimes.
arXiv Machine Learning
Jun 2
Riemannian Stochastic Optimization for Sufficient Dimension Reduction
arXiv:2606. 00413v1 Announce Type: cross Abstract: Sufficient dimension reduction (SDR) makes high-dimensional regression tractable by projecting the covariates onto a low-dimensional subspace that preserves the conditional mean of the response.
arXiv AI
Jun 16
SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation
arXiv:2606. 16454v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks by parameterizing weight updates with low-rank matrices.
arXiv Machine Learning
4d ago
Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression
arXiv:2602.02848v2 Announce Type: replace Abstract: Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD...
arXiv Machine Learning
Sep 14
Nonlinear Dimensionality Reduction Techniques for Bayesian Optimization
The paper investigates nonlinear dimensionality reduction for Bayesian optimisation (BO) by transforming high‑dimensional black‑box optimisation problems into a sequence of low‑dimensional latent‑space BO (LSBO) tasks. It extends earlier linear embedding approaches by using variational autoencoders (VAEs), deep metric loss, and adaptive retraining to better capture nonlinear structure, and couples LSBO with sequential domain reduction (SDR‑LSBO) to progressively narrow search domains. Experiments on GPU‑accelerated BoTorch with Matérn‑5/2 Gaussian‑process surrogates show that VAE‑based LSBO outperforms adaptive linear embeddings, and the authors provide a theoretical analysis of latent‑space error versus representation gap under PAC‑Bayes conditions.
arXiv AI
Aug 3
GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR
arXiv:2601. 09361v4 Announce Type: replace-cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is a key paradigm for improving large-scale reasoning models.
arXiv Machine Learning
Sep 17
Gradient Descent with Stochastic Subspaces via Persistence of Memory
The paper introduces a novel technique called "persistence of memory" to enhance stochastic subspace methods for large‑scale optimisation. By using a weakly correlated guidance vector that is refreshed only at wide intervals, the method provides a structured direction for random subspace descent. The authors demonstrate that this guidance can be efficiently computed in sparse or minibatch settings and present the first theoretical analysis of classical SSD methods for sparse functions, showing alignment with low‑lying Hessian eigenvectors near the optimum.
arXiv Machine Learning
Jun 11
A Riemannian Approach to Low-Rank Optimal Transport
arXiv:2606. 12120v1 Announce Type: new Abstract: Low-rank optimal transport (OT) mitigates the quadratic scaling of classical solvers, yet existing approaches rely heavily on first-order mirror-descent updates that require careful hyperparameter tuning and ignore the optimization landscape's curvature.
arXiv Machine Learning
Sep 21
Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blocks
arXiv:2609.21039v1 Announce Type: new Abstract: A pervasive structural pattern in modern deep learning is the linear factorization block: a submodule of the form $W = BA$ in which two parameter matri...
arXiv Machine Learning
Jul 7
Local Constrained Bayesian Optimization
arXiv:2603. 07965v2 Announce Type: replace-cross Abstract: Bayesian optimization (BO) for high-dimensional constrained problems remains a significant challenge due to the curse of dimensionality.
arXiv Machine Learning
Jul 23
A Bayesian Framework for Built-in Input Dimension Reduction for Gaussian Process Modeling
arXiv:2607. 19498v1 Announce Type: cross Abstract: Gaussian process (GP) modeling is widely used in computational science and engineering.