arXiv Machine Learning July 7, 2026 By Omer Tariq, Syed Muhammad Raza, Jeongbae Son

SAD-LoRA: Spectral Alignment for Low-Rank Knowledge Distillation

Read the original on arXiv Machine Learning →

arXiv:2607. 04306v1 Announce Type: new Abstract: Distilling a fine-tuned teacher into a LoRA-adapted student is a standard recipe for parameter-efficient compression, but output-level KD does not explicitly control which rank-$r$ weight subspace the adapter occupies.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 9

Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates

arXiv:2606. 07596v1 Announce Type: new Abstract: Fine-tuning often introduces spurious correlations alongside task knowledge, causing systematic failures on underrepresented groups.

By Edward Sun, Dmitrii Troitskii

fine-tuning benchmarks

arXiv Machine Learning

Jul 31

Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection

arXiv:2607. 27680v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models, yet its statistical properties remain only partially understood.

By Arunan J

fine-tuning benchmarks

arXiv AI

Jul 21

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

arXiv:2607. 16252v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models.

By Yupeng Chang, Yuan Wu, Yi Chang

llms fine-tuning benchmarks

arXiv AI

Jun 16

SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation

arXiv:2606. 16454v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks by parameterizing weight updates with low-rank matrices.

By Junghun Oh, Sungyong Baik, Kyoung Mu Lee

fine-tuning benchmarks safety

arXiv Machine Learning

Jul 7

CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation

arXiv:2607. 02576v1 Announce Type: cross Abstract: Parameter-Efficient Fine-Tuning (PEFT) commonly adapts pretrained weights through low-rank updates, and recent methods further exploit the singular value decomposition (SVD) of the base weight for initialization or subspace selection.

By Pengcheng Wang, Ziran Liu, Wei Wang, Wei Jiang

fine-tuning

arXiv AI

Jun 24

Weight-Space Geometry of Offline Reasoning Training

arXiv:2606. 23740v1 Announce Type: cross Abstract: Offline reinforcement-learning losses (RFT, RIFT, DFT, Offline GRPO, DPO) are widely used to distill reasoning from large teachers into smaller students, and are typically compared on downstream accuracy alone.

By Aleksandr Nikolich, Igor Kiselev, Vladimir Platonov, Karina Romanova

reinforcement-learning fine-tuning

Related stories

Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates

Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation

CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation

Weight-Space Geometry of Offline Reasoning Training