arXiv Computation and Language October 28, 2026 By Rong Huang, Yi Xiong, Yuan-Yuan Cheng, Xiao-Ming Fu

Hierarchical Low-Rank Reparameterization for Parameter-Efficient

Read the original on arXiv Computation and Language →

The paper introduces a hierarchical low‑rank reparameterization technique that reorganizes the trainable parameters of parameter‑efficient fine‑tuning (PEFT) methods into an approximately square matrix and applies recursive low‑rank decomposition. This multi‑level compression drastically reduces the number of trainable parameters while preserving strong task performance. Experiments on LLaMA‑3.1‑8B and Qwen2.5‑7B show that, when combined with LoRA or DoRA, the method outperforms baseline PEFT approaches and remains competitive even after compressing the original parameter count by an order of magnitude.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Jun 2

RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models

arXiv:2505. 18877v4 Announce Type: replace Abstract: Low-Rank Adaptation (LoRA) lowers the computational and memory overhead of fine-tuning large models by updating a low-dimensional subspace of the pre-trained weight matrix.

By Yilang Zhang, Bingcong Li, Georgios B. Giannakis

llms fine-tuning benchmarks

arXiv Machine Learning

Jul 30

Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation

arXiv:2510. 00192v3 Announce Type: replace Abstract: Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full fine-tuning.

By Xin Yu, Cong Xie, Xunmei Liu, Tiantian Fan, Lingzhou Xue, Zhi Zhang

llms fine-tuning efficiency

arXiv AI

Aug 7

Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging

arXiv:2608. 05541v1 Announce Type: new Abstract: Evolution Strategy (ES) is a promising alternative to gradient-based fine-tuning for resource-constrained Large Language Model (LLM) reasoning.

By Yu Gu, Zhi Zheng, Yunpeng Ba, Xialiang Tong, Mingxuan Yuan, Zhenkun Wang

llms fine-tuning

arXiv AI

Oct 23

NormTune: Norm-Guided Parameter-Efficient Fine-Tuning for Reasoning-Intensive Tasks

NormTune introduces a norm‑guided approach to parameter‑efficient fine‑tuning (PEFT) for reasoning‑intensive tasks, addressing a performance gap caused by under‑adaptation in weight‑update norms. By adaptively strengthening layers with insufficient updates, NormTune consistently improves various PEFT methods across challenging benchmarks. Experiments show significant accuracy gains—up to 24.24% with rank‑8 LoRA on S1K‑1.1—while rank‑256 variants match full fine‑tuning performance.

By Zhangming Li, Qinghao Hu, Weifan Guan, Yifan Zhang, Peisong Wang, Jian Cheng

fine-tuning benchmarks

arXiv AI

Aug 3

GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR

arXiv:2601. 09361v4 Announce Type: replace-cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is a key paradigm for improving large-scale reasoning models.

By Jiaying Zhang, Lei Shi, Jiguo Li, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He

llms reinforcement-learning fine-tuning

arXiv AI

Jul 21

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.

By Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing, Zhuo Zheng, Danning Ke, Qihong Lin, Qiang Wang, Shaohuai Shi, Xiaowen Chu

llms fine-tuning efficiency benchmarks

One email a morning, machine-written

Related stories

RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models

Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation

Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging

NormTune: Norm-Guided Parameter-Efficient Fine-Tuning for Reasoning-Intensive Tasks

GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models