arXiv Machine Learning
Principal-timestep Restricted Init via Sparse Matrix-decomposition in Flow-matching
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jun 16
SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation
arXiv:2606. 16454v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks by parameterizing weight updates with low-rank matrices.
arXiv Machine Learning
Oct 19
Ori: Drift-Aware Streaming $k$-PCA for Very Low-Rank LLM Training
arXiv:2610. 22794v1 Announce Type: new Abstract: A prominent family of low-rank LLM optimizers, pioneered by GaLore, performs gradient-subspace projection: at each step, they project the gradient into a low-dimensional subspace and map the update back to the full model.
arXiv AI
Jul 15
CARE-LoRA: Compressed Activation REconstruction for Memory-Efficient LoRA
arXiv:2607. 11940v1 Announce Type: cross Abstract: As the scale of large pre-trained models continues to grow, fine-tuning them under limited memory budgets has become increasingly challenging.
arXiv AI
Aug 21
LoRA-GA$^2$: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment
arXiv:2608. 19800v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) is a prominent fine-tuning method for large models, achieving competitive performance with reduced memory overhead.
arXiv AI
Jun 15
Beyond LoRA: Is Sparsity-Induced Adaptation Better?
arXiv:2606. 13767v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models.
arXiv Machine Learning
Jul 30
Between Gradient and Natural Gradient: A Continuum of LoRA Initializations
arXiv:2607. 26247v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized.