arXiv AI August 18, 2026 By Kasun Dewage, Marianna Pensky, Suranadi De Silva, Shankadeep Mondal

LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights

arXiv:2602. 17510v2 Announce Type: replace-cross Abstract: We introduce LoRA-CRAFT (\textbf{C}ross-layer \textbf{R}ank \textbf{A}daptation via \textbf{F}rozen \textbf{T}ucker), abbreviated CRAFT throughout, an extremely parameter-efficient fine-tuning (PEFT) method that applies Tucker tensor decomposition to pre-trained attention weight matrices stacked across transformer layers and trains only small square adaptation matrices on the resulting frozen Tucker factors.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv Machine Learning

Jul 27

Layer-wise LoRA fine-tuning: a similarity metric approach

arXiv:2602. 05988v2 Announce Type: replace Abstract: Pre-training Large Language Models (LLMs) on web-scale datasets becomes fundamental for advancing general-purpose AI.

By Keith Ando Ogawa, Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Lucas Pellicer, Rosimeire Pereira Costa, Edson Bollis, Anna Helena Reali Costa, Artur Jordao

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 2

Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters

arXiv:2606. 00428v1 Announce Type: cross Abstract: Low-rank adapters are usually compared by sweeping a small set of ranks, but the rank also fixes the resolution of the parameter budget.

By Xinjue Wang, Xiuheng Wang, Yejun Zhang, Sergiy A. Vorobyov, Esa Ollila, Zhi-Yong Wang

fine-tuning

arXiv AI

Jun 15

Beyond LoRA: Is Sparsity-Induced Adaptation Better?

arXiv:2606. 13767v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models.

By Elijah Cadenhead, Cristian McGee, Xin Li, El Houcine Bergou, Aritra Dutta

fine-tuning efficiency

arXiv AI

Jul 21

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

arXiv:2607. 16252v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models.

By Yupeng Chang, Yuan Wu, Yi Chang

llms fine-tuning benchmarks

arXiv Machine Learning

Jul 30

Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

arXiv:2607. 26247v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized.

By Dianze Liu, Farshid Ghezelbash

llms fine-tuning

arXiv Machine Learning

Jul 21

PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer

arXiv:2607. 17620v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) makes finetuning large language models cheaper by adding to each weight matrix a trainable low-rank update parameterized as the product of two matrices.

By Nikhil Ghosh, Tetiana Parshakova, Robert M. Gower

llms fine-tuning

Related stories

Layer-wise LoRA fine-tuning: a similarity metric approach

Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters

Beyond LoRA: Is Sparsity-Induced Adaptation Better?

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer