arXiv:2606. 29184v1 Announce Type: new Abstract: While Low-rank adaptation (LoRA) enables highly efficient fine-tuning by constraining task-specific updates to fixed low-rank subspaces, this rigid design limits representational flexibility and often results in overconfident predictions and miscalibrated uncertainty, especially in low-data regimes.
arXiv:2510. 24561v3 Announce Type: replace-cross Abstract: LoRA has become a widely adopted method for PEFT, and its initialization methods have attracted increasing attention.
arXiv:2601. 21003v3 Announce Type: replace Abstract: Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration.
TaRA: Training-Aware Low-Rank Adaptation Initialization proposes a new way to initialize LoRA by aligning the gradients of low‑rank factors with those of the full‑rank weight matrix. This approach directly incorporates training dynamics, improving gradient fidelity at the start of fine‑tuning while adding negligible computational cost. Experiments on a variety of challenging fine‑tuning tasks show that TaRA consistently outperforms existing state‑of‑the‑art initialization methods, offering a simple, robust, and scalable solution for effective LoRA initialization.
arXiv:2607. 27680v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models, yet its statistical properties remain only partially understood.
arXiv:2606. 13767v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models.
arXiv:2607. 02182v1 Announce Type: new Abstract: Large language models (LLMs) exhibit remarkable reasoning capabilities, but their task-specific fine-tuning is notoriously plagued by overconfidence, severely hindering trustworthy deployment.
The study investigates whether coefficient priors derived from large language models (LLMs) can enhance out‑of‑distribution (OOD) tabular prediction by serving as a shrinkage target in regularised regression. Across five datasets, LLM‑elicited priors improved OOD mean squared error on four datasets by up to 7.9% and yielded better probabilistic accuracy in classification tasks, though the gains varied and were sometimes comparable to in‑domain improvements. Cross‑validation was used to select priors, but the variability among candidates meant that the method mainly avoided the worst priors rather than consistently finding the best. "whyItMatters":"The findings suggest that LLM‑derived priors can provide a scalable alternative to expert elicitation for improving model robustness to distribution shifts, though their effectiveness depends on dataset size and prior variability."
arXiv:2602. 00387v4 Announce Type: replace-cross Abstract: Bayesian neural networks promise calibrated uncertainty but require $O(mn)$ parameters for standard mean-field Gaussian posteriors.
arXiv:2607. 27023v1 Announce Type: new Abstract: Evaluating large generative models across benchmarks is time-consuming and computationally expensive.
arXiv:2608. 20061v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures significantly expand model capacity without a proportional increase in computational cost.
arXiv:2507.23768v2 Announce Type: replace-cross Abstract: Existing methods for transfer learning struggle to deal with situations where the source datasets are limited and not guaranteed to be well-a...