arXiv AI July 1, 2026 By Ruijia Zhang, Jiacheng Zhu, Hanqing Zhu, Laixi Shi

Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

arXiv:2606. 31813v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning (SFT) paradigm.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

Aug 3

GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR

arXiv:2601. 09361v4 Announce Type: replace-cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is a key paradigm for improving large-scale reasoning models.

By Jiaying Zhang, Lei Shi, Jiguo Li, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He

llms reinforcement-learning fine-tuning

arXiv AI

Jul 21

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

arXiv:2607. 16252v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models.

By Yupeng Chang, Yuan Wu, Yi Chang

llms fine-tuning benchmarks

arXiv AI

Jun 8

On the Geometry of On-Policy Distillation

arXiv:2606. 07082v1 Announce Type: cross Abstract: On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood.

By Zhennan Shen, Yanshu Li, Qingyu Yin, Chak Tou Leong, Zhilin Wang, Yanxu Chen, Rongduo Han, Sunbowen Lee, Yi R. Fung

llms reinforcement-learning fine-tuning efficiency

arXiv Machine Learning

Jul 30

Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

arXiv:2607. 26247v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized.

By Dianze Liu, Farshid Ghezelbash

llms fine-tuning

arXiv Machine Learning

Jul 23

The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability

arXiv:2607. 20301v1 Announce Type: new Abstract: Fine-tuning has been widely used to adapt large language models (LLMs) for domain-specific tasks.

By Abigail Woodring, Adrian Chan, Rana Muhammad Shahroz Khan, Sukwon Yun, Chau-Wai Wong, Tianlong Chen

llms fine-tuning

arXiv AI

Jun 16

SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation

arXiv:2606. 16454v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks by parameterizing weight updates with low-rank matrices.

By Junghun Oh, Sungyong Baik, Kyoung Mu Lee

fine-tuning benchmarks safety

Related stories

GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

On the Geometry of On-Policy Distillation

Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability

SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation