arXiv Machine Learning
Aug 27
How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention
arXiv:2608. 26052v1 Announce Type: new Abstract: Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task.
The Approximation Rank of Softmax Attention: Sharp Geometric Laws and Robust Interaction Dimension
arXiv:2608. 28150v1 Announce Type: new Abstract: Which geometry controls the rank complexity of normalized softmax attention?
Related stories
arXiv Machine Learning
Jul 28
The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers
arXiv:2607. 23050v1 Announce Type: new Abstract: Neural scaling laws describe how loss decreases as models, data, and compute grow, but they do not answer a prior question: for a fixed task, what is the minimum model capacity required to solve it?
arXiv AI
Aug 18
Exact Attention Sensitivity and the Geometry of Transformer Stability
arXiv:2602. 18849v2 Announce Type: replace-cross Abstract: We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation.
arXiv Computation and Language
2d ago
Query-Oblivious Coresets for Softmax Attention: Improved Bounds and Efficient Constructions
arXiv:2609.06327v2 Announce Type: replace-cross Abstract: A query-oblivious coreset for a softmax-attention head is a subset of the key-value pairs whose attention output is within $\varepsilon$ of t...
arXiv Machine Learning
6d ago
Toward a First-Principles Update Geometry for the Language-Model Head
The paper proposes a new update geometry for the language‑model head by treating the head and softmax as a single module and using Hilbert’s projective distance to measure functional change. It replaces the spectral norm with the Euclidean row diameter, derives a tractable RowNorm update rule, and demonstrates that RowNorm substantially reduces step diameters and Hilbert perturbations while only slightly increasing validation loss.
arXiv Machine Learning
Sep 10
Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks
arXiv:2609.09130v1 Announce Type: new Abstract: An input may activate few hidden units even when different inputs collectively use an entire network. We study the statistical complexity of this input...
arXiv Machine Learning
Jul 22
Geometric Capacity of Transformers: A Tropical Geometry Perspective
arXiv:2604. 14727v2 Announce Type: replace Abstract: To quantify the geometric capacity of transformers, we develop a tropical-geometric framework for analyzing the spatial partitions induced by conditioned self-attention.
Hugging Face Trending Papers
Sep 8
Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks
An input may activate few hidden units even when different inputs collectively use an entire network. We study the statistical complexity of this input-dependent sparsity in the one-hidden-layer ReLU model of Awasthi et al.
arXiv Machine Learning
Aug 11
Training-Free Universal Approximation by Prompting Random Transformers
arXiv:2608. 09558v1 Announce Type: new Abstract: How expressive is prompting a transformer?
arXiv AI
Jun 3
$\mathbb{R}^{2k}$ is Theoretically Large Enough for Embedding-based Top-$k$ Retrieval
arXiv:2601. 20844v3 Announce Type: replace-cross Abstract: This paper studies the Minimal Embeddable Dimension (MED): the least dimension in which there exists a configuration of $m$ object vectors so that every subset of size at most $k$ is exactly retrieved by score comparison.
arXiv Machine Learning
Sep 2
Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras
arXiv:2609.01129v1 Announce Type: new Abstract: We identify a recurrent algebraic regularity in Transformer attention: a sparse subset of effective OV operators $T=OV^\top$ nearly closes under compos...
Hugging Face Trending Papers
Aug 11
Three Tokens Force Exponential Feature Rank in Nonnegative Kernel Attention
Full attention exposes every token pair, whereas kernel attention compresses a sequence into a fixed-dimensional sketch. We show that this distinction becomes exponential at the first context length containing two competing candidates.