arXiv:2608. 26052v1 Announce Type: new Abstract: Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task.
arXiv:2607. 23050v1 Announce Type: new Abstract: Neural scaling laws describe how loss decreases as models, data, and compute grow, but they do not answer a prior question: for a fixed task, what is the minimum model capacity required to solve it?
arXiv:2609.32002v2 Announce Type: replace Abstract: The rank $r$ in LoRA is widely treated as a capacity control: a smaller rank is assumed to yield a simpler model that generalizes better. We show t...
arXiv:2610.19033v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) applies every rank-one component of its update to every token, and mixture-of-experts variants replace this with a learned r...
arXiv:2610.23343v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) has become the standard method for parameter-efficient fine-tuning of large pretrained models, yet theoretical explanations...
arXiv:2609.21523v1 Announce Type: new Abstract: A system may be compressed before its downstream task is fully known. We ask how much retained state is then necessary and how much can be saved by lim...
arXiv:2608. 28150v1 Announce Type: new Abstract: Which geometry controls the rank complexity of normalized softmax attention?
The paper introduces a low‑rank framework for ranking large language models (LLMs) on task‑specific benchmarks using sparse pairwise comparisons. By modeling the task‑by‑model ability matrix as low rank, the method shares information across related tasks while preserving task‑specific differences, and it provides uncertainty‑aware ranking through debiased estimators and simultaneous confidence sets. Experiments on synthetic data and the Chatbot Arena benchmark demonstrate improved sample efficiency and tighter, better‑calibrated ranking certificates, especially in the sparse comparison regime typical of real LLM evaluations.
arXiv:2608.28150v2 Announce Type: replace Abstract: How much matrix rank is required to preserve every bounded value output of normalized softmax attention? We study the unrestricted maximum-row-\(\e...
arXiv:2609.36766v1 Announce Type: new Abstract: Can a fixed continuous prefix replace a given low-rank adapter while the attention head stays frozen? In this research, we show that the answer depends...
The paper introduces Outlier-Aware Attention Linearization (OAL), a method that improves linearized softmax attention by explicitly accounting for a few outlier dimensions that disproportionately affect query-key dot products. OAL builds on a second-order Taylor approximation, uses a Hadamard-product formulation to expose dimension-specific contributions, and groups dimensions with similar scales to share coefficients while treating outliers separately. Experiments on Qwen2.5-0.5B show an 8.9% reduction in frozen-attention output MSE, and fine-tuning with LoRA lowers WikiText-103 perplexity from 31.71 to 27.50 while boosting performance on PIQA, HellaSwag, and ARC-E.
arXiv:2607. 27680v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models, yet its statistical properties remain only partially understood.
arXiv:2607. 20205v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models.