arXiv AI
Aug 3
GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR
arXiv:2601. 09361v4 Announce Type: replace-cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is a key paradigm for improving large-scale reasoning models.
Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR
arXiv:2606. 31813v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning (SFT) paradigm.
Related stories
arXiv AI
Jul 21
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
arXiv:2607. 16252v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models.
arXiv AI
Jun 8
On the Geometry of On-Policy Distillation
arXiv:2606. 07082v1 Announce Type: cross Abstract: On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood.
arXiv Machine Learning
Jul 30
Between Gradient and Natural Gradient: A Continuum of LoRA Initializations
arXiv:2607. 26247v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized.
arXiv Machine Learning
Jul 23
The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability
arXiv:2607. 20301v1 Announce Type: new Abstract: Fine-tuning has been widely used to adapt large language models (LLMs) for domain-specific tasks.
arXiv AI
Jun 16
SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation
arXiv:2606. 16454v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks by parameterizing weight updates with low-rank matrices.
arXiv AI
6d ago
GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs
arXiv:2608. 11674v1 Announce Type: cross Abstract: On-policy rollout methods such as GRPO are central to post-training of large language models, yet they frequently suffer from training instabilities, cross-task capability degradation, and response-length inflation.
arXiv Machine Learning
Aug 4
Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models
arXiv:2608. 01624v1 Announce Type: cross Abstract: Adapting a language model to a task no longer requires training all of its weights, and a line of parameter-efficient methods has driven the trainable count from billions down to a handful of scalars.
arXiv Machine Learning
Jun 2
RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
arXiv:2505. 18877v4 Announce Type: replace Abstract: Low-Rank Adaptation (LoRA) lowers the computational and memory overhead of fine-tuning large models by updating a low-dimensional subspace of the pre-trained weight matrix.
arXiv AI
Aug 7
Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging
arXiv:2608. 05541v1 Announce Type: new Abstract: Evolution Strategy (ES) is a promising alternative to gradient-based fine-tuning for resource-constrained Large Language Model (LLM) reasoning.
arXiv Machine Learning
Jul 27
IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning
arXiv:2607. 22251v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method for large language models, but its performance depends strongly on how a fixed rank budget is distributed across Transformer modules.
arXiv Machine Learning
Jul 30
Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation
arXiv:2510. 00192v3 Announce Type: replace Abstract: Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full fine-tuning.