arXiv Statistics ML
Prior Predictive Matching for Bayesian LoRA
Read the original on arXiv Statistics ML →The Flow has not summarised this story yet — read it at arXiv Statistics ML.
Related stories
arXiv Machine Learning
Jun 30
BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning
arXiv:2606. 29184v1 Announce Type: new Abstract: While Low-rank adaptation (LoRA) enables highly efficient fine-tuning by constraining task-specific updates to fixed low-rank subspaces, this rigid design limits representational flexibility and often results in overconfident predictions and miscalibrated uncertainty, especially in low-data regimes.
arXiv AI
Jun 8
LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis
arXiv:2510. 24561v3 Announce Type: replace-cross Abstract: LoRA has become a widely adopted method for PEFT, and its initialization methods have attracted increasing attention.
arXiv AI
Jul 28
Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
arXiv:2601. 21003v3 Announce Type: replace Abstract: Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration.
arXiv AI
Sep 3
TaRA: Training-Aware Low-Rank Adaptation Initialization
TaRA: Training-Aware Low-Rank Adaptation Initialization proposes a new way to initialize LoRA by aligning the gradients of low‑rank factors with those of the full‑rank weight matrix. This approach directly incorporates training dynamics, improving gradient fidelity at the start of fine‑tuning while adding negligible computational cost. Experiments on a variety of challenging fine‑tuning tasks show that TaRA consistently outperforms existing state‑of‑the‑art initialization methods, offering a simple, robust, and scalable solution for effective LoRA initialization.
arXiv Machine Learning
Jul 31
Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection
arXiv:2607. 27680v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models, yet its statistical properties remain only partially understood.
arXiv AI
Jun 15
Beyond LoRA: Is Sparsity-Induced Adaptation Better?
arXiv:2606. 13767v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models.