arXiv Machine Learning
How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Experts
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Oct 3
Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Experts
The paper introduces VANE, a new routing strategy for Mixture-of-LoRA-experts that scores experts based on the expected loss reduction of their updates rather than token similarity. By decomposing each expert into a reader and writer, VANE evaluates all reader–writer pairs using a low‑rank compass that predicts the descent direction, enabling efficient top‑k selection with additive gates. Experiments on Llama‑3 models show VANE outperforms twelve PEFT and MoE‑LoRA baselines while using fewer trainable parameters and providing router scores that better reflect expert usefulness.
arXiv Machine Learning
Aug 4
Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA
arXiv:2607. 26052v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) variants of Low-Rank Adaptation (LoRA) route every token to a fixed number of experts $k$.
arXiv Machine Learning
Sep 29
New LoRA Skills Should Read but Never Write
The paper introduces READ, a method for composing low‑rank adapters (LoRA) in large language models. By rewriting each adapter into a balanced canonical form and enforcing a one‑directional coupling, READ allows new skills to read but never write into the output subspaces of existing skills, eliminating interference. Experiments on four benchmark suites and two model families show that READ consistently outperforms existing baselines, improving SuperGLUE scores by over twenty points and domain suite scores by more than seven points.
arXiv Machine Learning
Aug 27
How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention
arXiv:2608. 26052v1 Announce Type: new Abstract: Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task.
arXiv Machine Learning
Jul 31
Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection
arXiv:2607. 27680v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models, yet its statistical properties remain only partially understood.
arXiv AI
Oct 21
What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation
arXiv:2609.32002v2 Announce Type: replace Abstract: The rank $r$ in LoRA is widely treated as a capacity control: a smaller rank is assumed to yield a simpler model that generalizes better. We show t...