arXiv Machine Learning
Intrinsic Interaction Geometry Controls the Low-Rank Complexity of Softmax Attention
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Aug 31
The Approximation Rank of Softmax Attention: Sharp Geometric Laws and Robust Interaction Dimension
arXiv:2608. 28150v1 Announce Type: new Abstract: Which geometry controls the rank complexity of normalized softmax attention?
arXiv AI
Aug 18
Exact Attention Sensitivity and the Geometry of Transformer Stability
arXiv:2602. 18849v2 Announce Type: replace-cross Abstract: We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation.
arXiv AI
Jul 16
ExTernD: Expanded-Rank Ternary Decomposition Ternary LLM PTQ with Accuracy Approaching Any Quantization Level
arXiv:2607. 13511v1 Announce Type: cross Abstract: We introduce ExTernD (Expanded-rank Ternary Decomposition), a post-training factorization of each LLM weight matrix $A \in \mathbb{R}^{m \times n}$ into $A \approx B \mathrm{diag}(D) C$ with ternary factors $B \in \{-1,0,+1\}^{m \times k}$, $C \in \{-1,0,+1\}^{k \times n}$ and a real scale vector $D \in \mathbb{R}^k$.
arXiv Machine Learning
Jul 28
The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers
arXiv:2607. 23050v1 Announce Type: new Abstract: Neural scaling laws describe how loss decreases as models, data, and compute grow, but they do not answer a prior question: for a fixed task, what is the minimum model capacity required to solve it?
arXiv Machine Learning
Aug 27
How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention
arXiv:2608. 26052v1 Announce Type: new Abstract: Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task.
arXiv Computation and Language
Sep 15
Query-Oblivious Coresets for Softmax Attention: Improved Bounds and Efficient Constructions
arXiv:2609.06327v2 Announce Type: replace-cross Abstract: A query-oblivious coreset for a softmax-attention head is a subset of the key-value pairs whose attention output is within $\varepsilon$ of t...