arXiv Machine Learning
Aug 6
Attention-based representations for multi-task computation
arXiv:2608. 04243v1 Announce Type: new Abstract: Multi-head attention layers produce vector representations that support multiple downstream tasks.
The Head Complexity of Boolean Functions in Single-Layer Attention
arXiv:2609. 04046v1 Announce Type: cross Abstract: What can a single layer of self-attention compute?
Related stories
arXiv Machine Learning
Sep 22
Counting and Covering in Nearest-Neighbour Representations of Boolean Functions
arXiv:2609.23094v1 Announce Type: cross Abstract: We study the number of prototypes needed to represent Boolean functions by nearest-neighbour classification. There are two distinct settings: the pro...
arXiv Machine Learning
Sep 1
Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers
arXiv:2608.31067v1 Announce Type: new Abstract: Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and lengt...
arXiv Machine Learning
Jun 17
Tight $L_\infty$ Sample Complexity for Low-Degree and Sparse Boolean Polynomials
arXiv:2606. 17319v1 Announce Type: cross Abstract: Motivated by the optimization of bounded binary black-box functions, we study the problem of learning polynomial surrogates over the Boolean hypercube.
arXiv Machine Learning
Oct 21
Communication Complexity of Universal Features
arXiv:2610.27402v1 Announce Type: cross Abstract: Motivated by the notion of features from machine learning, we study the complexity of representing data that allows for evaluation with a given famil...
arXiv Machine Learning
Jun 11
Higher-Order Token Interactions via Quantum Attention
arXiv:2606. 11673v1 Announce Type: cross Abstract: Standard dot-product self-attention computes, in a single layer, only pairwise (order-2) interactions between tokens; representing a generic order-$k$ interaction is known to require either super-quadratic resources in one layer or composition across depth.
arXiv Machine Learning
Aug 4
Do Neural Networks Really Beat the Curse of Dimensionality? A Bit-Complexity View
arXiv:2608. 01357v1 Announce Type: new Abstract: Traditional approximation theory measures convergence rates in terms of the number of parameters or degrees of freedom.
Hugging Face Trending Papers
Aug 3
Optimal Unambiguous DNFs and Alon-Saks-Seymour
We construct unambiguous DNFs having width $O(n)$ but $0$-certificate complexity $Ω(n^2)$. By utilizing the special structure of these DNFs, we prove a lifting theorem with a constant-sized gadget that lifts the DNF to a communication problem, while losslessly translating the separation in certificate complexity to a separation in communication complexity.
arXiv Machine Learning
Sep 30
Efficiently Approximating Attention Is Hard
arXiv:2609.37261v1 Announce Type: new Abstract: Softmax attention is ubiquitous in modern machine learning, but its quadratic scaling with sequence length makes it costly. To reduce this cost, attent...
arXiv Machine Learning
Aug 13
Chain-of-Thought Shows the Path to a Tree: Realizing Branching Complexity
arXiv:2608. 11716v1 Announce Type: new Abstract: Chain of Thought (CoT) lifts the expressive ceiling of bounded-depth Transformers, with characterizations tying the number of CoT steps to circuit complexity classes.
arXiv Machine Learning
Jul 27
Indexing: the Beginning and the End
arXiv:2607. 22361v1 Announce Type: new Abstract: We study information bottlenecks in modern deep-learning architectures -- RNNs, softmax transformers, linear-attention transformers and state-space models -- through the lens of the indexing primitive.
arXiv Machine Learning
Aug 4
Optimal Unambiguous DNFs and Alon-Saks-Seymour
arXiv:2608. 02533v1 Announce Type: cross Abstract: We construct unambiguous DNFs having width $O(n)$ but $0$-certificate complexity $\Omega(n^2)$.