arXiv Machine Learning July 7, 2026 By Pengcheng Wang, Ziran Liu, Wei Wang, Wei Jiang

CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation

Read the original on arXiv Machine Learning →

arXiv:2607. 02576v1 Announce Type: cross Abstract: Parameter-Efficient Fine-Tuning (PEFT) commonly adapts pretrained weights through low-rank updates, and recent methods further exploit the singular value decomposition (SVD) of the base weight for initialization or subspace selection.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

fine-tuning

arXiv AI

Jul 1

Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

arXiv:2606. 31813v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning (SFT) paradigm.

By Ruijia Zhang, Jiacheng Zhu, Hanqing Zhu, Laixi Shi

llms reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 7

SAD-LoRA: Spectral Alignment for Low-Rank Knowledge Distillation

arXiv:2607. 04306v1 Announce Type: new Abstract: Distilling a fine-tuned teacher into a LoRA-adapted student is a standard recipe for parameter-efficient compression, but output-level KD does not explicitly control which rank-$r$ weight subspace the adapter occupies.

By Omer Tariq, Syed Muhammad Raza, Jeongbae Son

fine-tuning efficiency safety

arXiv AI

Jul 21

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

arXiv:2607. 16252v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models.

By Yupeng Chang, Yuan Wu, Yi Chang

llms fine-tuning benchmarks

arXiv AI

1d ago

LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights

arXiv:2602. 17510v2 Announce Type: replace-cross Abstract: We introduce LoRA-CRAFT (\textbf{C}ross-layer \textbf{R}ank \textbf{A}daptation via \textbf{F}rozen \textbf{T}ucker), abbreviated CRAFT throughout, an extremely parameter-efficient fine-tuning (PEFT) method that applies Tucker tensor decomposition to pre-trained attention weight matrices stacked across transformer layers and trains only small square adaptation matrices on the resulting frozen Tucker factors.

By Kasun Dewage, Marianna Pensky, Suranadi De Silva, Shankadeep Mondal

llms fine-tuning benchmarks

arXiv Machine Learning

Aug 6

The Loss Does Not See the Basis, but Adam Does

arXiv:2608. 05136v1 Announce Type: new Abstract: Gradient descent on a factored model $W = UV^\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not.

By Devender Singh

llms safety

arXiv Machine Learning

Jun 9

Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates

arXiv:2606. 07596v1 Announce Type: new Abstract: Fine-tuning often introduces spurious correlations alongside task knowledge, causing systematic failures on underrepresented groups.

By Edward Sun, Dmitrii Troitskii

fine-tuning benchmarks

Related stories

Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

SAD-LoRA: Spectral Alignment for Low-Rank Knowledge Distillation

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights

The Loss Does Not See the Basis, but Adam Does

Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates