arXiv Machine Learning June 30, 2026 By Sahar Rajabi, Nayeema Nonta, Sirisha Rambhatla

Geometrically Principled Randomized Optimization for Efficient LLM Training

Read the original on arXiv Machine Learning →

arXiv:2510. 01878v2 Announce Type: replace Abstract: Low-rank gradient optimization for large language models is currently divided into two categories: structured methods that rigorously identify subspaces, and randomized approaches employed primarily for computational efficiency.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 2

Riemannian Stochastic Optimization for Sufficient Dimension Reduction

arXiv:2606. 00413v1 Announce Type: cross Abstract: Sufficient dimension reduction (SDR) makes high-dimensional regression tractable by projecting the covariates onto a low-dimensional subspace that preserves the conditional mean of the response.

By Thibault Pautrel, Fran\c{c}ois Portier

arXiv AI

Jul 29

Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning

arXiv:2607. 25299v1 Announce Type: cross Abstract: Optimization over the Stiefel manifold plays a significant role in various machine learning tasks.

By Yuan Zhang, Jiang Hu, Zhijian Lai, Lin Lin, Zaiwen Wen

llms fine-tuning benchmarks

arXiv Machine Learning

Aug 6

Random features for Grassmannian kernel approximation with bounded rank-one projections

arXiv:2608. 04227v1 Announce Type: new Abstract: We propose a family of random feature maps for scalable kernel machines on low-dimensional subspaces, ie on the Grassmannian manifold.

By R\'emi Delogne, Laurent Jacques

rag

arXiv AI

Jul 1

Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

arXiv:2606. 31813v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning (SFT) paradigm.

By Ruijia Zhang, Jiacheng Zhu, Hanqing Zhu, Laixi Shi

llms reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 21

Gradient Span Algorithms Make Predictable Progress in High Dimension

arXiv:2410. 09973v2 Announce Type: replace-cross Abstract: We prove that all 'gradient span algorithms' have asymptotically deterministic behavior on scaled Gaussian random functions as the dimension tends to infinity.

By Felix Benning, Leif D\"oring

arXiv Machine Learning

Jun 11

Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning

arXiv:2601. 04710v2 Announce Type: replace-cross Abstract: Fine-tuning large language models (LLMs) achieves strong performance but is often limited by the memory overhead of backpropagation.

By Feihu Jin, Shipeng Cen, Ying Tan

llms fine-tuning benchmarks

Related stories

Riemannian Stochastic Optimization for Sufficient Dimension Reduction

Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning

Random features for Grassmannian kernel approximation with bounded rank-one projections

Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

Gradient Span Algorithms Make Predictable Progress in High Dimension

Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning