arXiv Machine Learning
LoRASpace: A Pool-Wide Shared Substrate for Static and Dynamic Multi-LoRA Composition
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 3
Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter
arXiv:2606. 03723v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) enables parameter-efficient specialization of foundation models, but the proliferation of task-specific adapters fragments capabilities across many adapters, complicating reuse and deployment.
arXiv AI
Aug 18
LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights
arXiv:2602. 17510v2 Announce Type: replace-cross Abstract: We introduce LoRA-CRAFT (\textbf{C}ross-layer \textbf{R}ank \textbf{A}daptation via \textbf{F}rozen \textbf{T}ucker), abbreviated CRAFT throughout, an extremely parameter-efficient fine-tuning (PEFT) method that applies Tucker tensor decomposition to pre-trained attention weight matrices stacked across transformer layers and trains only small square adaptation matrices on the resulting frozen Tucker factors.
arXiv AI
Aug 11
Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach
arXiv:2608. 09742v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) represents large language model (LLM) updates with two compact matrix factors, i.
arXiv AI
Jun 15
Beyond LoRA: Is Sparsity-Induced Adaptation Better?
arXiv:2606. 13767v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models.
arXiv Machine Learning
Sep 14
Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature
The paper introduces ISO-LoRA, an optimizer that improves rank utilization in Low‑Rank Adaptation (LoRA) by coupling factor updates through spectral descent on the induced tangent perturbation in weight space. Experiments on GPT‑2 adaptation show that standard optimizers like AdamW concentrate updates in a few singular directions, whereas ISO-LoRA distributes energy more evenly, leading to higher effective rank and better downstream performance across 0.1B‑7B models. The authors provide theoretical guarantees under a stylized spiked‑gradient model and demonstrate that ISO-LoRA consistently outperforms factor‑wise optimizers, especially at moderate‑to‑large LoRA ranks.
arXiv Machine Learning
Sep 3
LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates
LoRA-TSD introduces a new optimizer for low‑rank adaptation (LoRA) that treats each update as a tangent vector on the fixed‑rank matrix manifold and applies a Muon‑style spectral‑norm steepest‑descent step within that tangent space. The method avoids costly full‑matrix operations and offers a retraction that is up to 2.8× cheaper than previous manifold approaches. The authors prove that their surrogate recovers LoRA‑Pro, identify the Riemannian gradient as the natural stationarity measure, and provide the first global convergence guarantees for both LoRA‑Pro and LoRA‑TSD, achieving superior performance across multiple benchmarks with Llama and Qwen models.